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SECTION  I 
INTRODUCTION 

The  objective  of  this  contract  is  to  begin  development  of  a  unified 
approach  to  the  important  problem  of  selecting  an  optimal  hardware  design 
for,  and  optimally  allocating  software  tasks  in,  avionic  information  pro¬ 
cessing  systems.  This  report  summarizes  the  technical  achievements  accom¬ 
plished  during  the  contractual  period. 

The  research  conducted  under  this  contract,  and  its  predecessor  [17], 
has  focused  on  methodology  for  the  combined  software  allocation/hardware 
design  problem.  The  two  aspects  of  the  problem  are  inherently  linked  in 
that  a  hardware  design  cannot  be  evaluated  properly  until  the  software 
allocation  is  specified,  and,  conversely,  that  the  "best"  location  for  the 
software  depends  critically  upon  the  hardware  design. 

The  first  step  by  Systems  Control  Technology  (SCT)  has  been  to  look 
at  the  two  aspects  of  the  problem  separately.  In  [17]  the  focus  was  on 
software  allocation.  There,  spatial  dynamic  programming  (SDP)  was  used 
to  solve  a  static,  deterministic  software  allocation  problem.  Under  the 
current  contract  the  SDP  methodology  has  been  extended  to  incorporate  both 
dynamic  an.'"  'ichastic  effects.  In  addition,  an  approach  for  evaluating 
software  allocations  in  a  stochastic  environment  has  been  developed  which 
is  quite  flexible,  yet  simple  and  computationally  tractable. 

Investigation  of  the  hardware  design  problem  has  also  been  initiated. 
A  method  based  on  graph  theory  has  been  used  to  evaluate  candidate  network 
architectures.  Specific  architectures  have  been  identified  which  exhibit 
extremal  values  of  important  attributes  such  as  maximum  interprocessor 
distance.  A  more  realistic  hardware  model  involving  buses  over  which 
several  processors  can  communicate  has  also  been  studied. 

Given  these  noteworthy  and  promising  developments  from  our  research, 
the  next  step  is  to  merge  tha.  software  allocation  and  hardware  design 
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aspects  of  the  problem  to  achieve  the  overall  goal  of  the  research  program. 
In  particular,  for  a  specific  set  of  software  functions  the  optimal  design 
and  optimal  allocation  must  be  identified  for  given  tradeoffs  involving 
cost,  performance,  and  availability.  We  anticipate  that  the  results  of 
this  contract  will  be  an  integral  part  of  the  solution  methodology  for  the 
combined  problem. 

This  report  is  organized  as  follows.  Section  2  addresses  the  software 
allocation  problem.  Each  of  the  subsections  here  focuses  on  a  separate 
aspect  of  the  problem.  Section  2.1  introduces  the  software  allocation 
problem  in  some  detail  and  also  discusses  the  computational  complexity 
of  the  SDP  algorithm.  Section  2.2  examines  several  methods  for  attacking 
the  software  allocation  problem  when  the  tasks  are  constrained  by  precedence 
relationships,  as  indeed  they  are  in  most  avionic  systems.  Section  2.3 
presents  a  methodology  for  evaluating  allocations  in  the  presence  of 
stochastic  fluctuations,  using  the  methods  of  stochastic  network  theory. 
Section  2.4  discusses  two  extensions  of  the  SDP  methodology  resulting  in  a 
more  accurate  model  -  one  involving  the  allocation  of  files  as  well  as 
tasks,  the  other  allowing  redundant  assignment  of  tasks  because  of  the 
possibility  of  hardware  failures. 

Section  3  concentrates  on  the  hardware  design  problem.  Section  3.1 
presents  an  overview  of  the  hardware  design  problem.  In  Section  3.2  a 
computer  program  called  NETEV  is  described;  this  program  has  been  used  to 
evaluate  candidate  architectures.  Section  3.3  discusses  a  particularly 
important  problem  in  the  design  of  networks  which  relates  to  minimizing 
the  maximum  inter-processor  distance.  Section  3.4  looks  at  the  problem 
of  network  reliability,  and  the  tradeoff  between  reliability  and  small 
distances.  Finally,  Section  3.5  examines  the  issue  of  how  to  design 
networks  when  buses  connecting  several  processors  are  available. 

The  report  also  includes  three  appendices.  Appendix  A  is  a  listing 
of  the  main  allocation  program  using  SDP;  a  sample  set  of  input  and  output 
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is  also  provided.  Appendix  B  is  a  listing  of  the  NETEV  program  used  to 
evaluate  candidate  computer  network  architectures.  Appendix  C  provides 
a  proof  for  a  theorem  presented  in  Section  3.4  involving  processors  with 
two  ports. 

Future  Directions 


The  research  described  here  is  a  first  step  to  achieving  the  important 
goal  of  a  complete  software  allocation/hardware  design  methodology  to 
significantly  improve  the  performance  of  avionic  information  processing 
systems.  Several  technical  approaches  for  moving  toward  this  unified 
methodology  are  currently  envisioned.  One  of  the  simplest  is  a  successive 
approximation  method.  For  a  given  hardware  arrangement,  the  optimal  soft¬ 
ware  allocation  is  identified,  perhaps  by  a  modification  of  the  SDP  algorithm 
in  which  variable  hardware  connections  are  allowed.  Given  the  optimal 
software  allocation,  the  optimal  hardware  connections  are  then  found. 

The  second  step  might  be  based  on  the  work  of  Torng  and  Wilhelm  [48],  and 
may  use  results  from  Section  3  of  this  report.  For  the  new  hardware  con¬ 
nection  scheme,  the  optimal  software  allocation  is  found,  and  the  process 
is  repeated  until  the  scheme  converges. 

Another  approach  is  to  blend  the  task-to-processor  assignment  graph 
with  the  bipartite  hardware  graph  of  Section  3.5.  The  result  would  be  a 
tripartite  graph,  consisting  of  tasks,  processors,  and  buses.  Tasks  would 
be  assigned  to  processors,  and  processors  assigned  to  buses.  The  solution 
method  could  again  be  based  on  SDP. 

Whatever  the  solution  method,  solving  the  unified  problem  will  produce 
significant  payoffs.  For  example,  the  effect  on  an  entire  system  of  adding 
another  processor  can  be  investigated.  Only  if  the  advantages  of  improved 
performance  and  availability  outweigh  the  increased  cost  should  that 
processor  be  added.  The  capability  for  solving  such  allocation  problems 
should  indeed  result  in  better  avionic  systems. 
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The  methods  developed  should  have  a  significant  impact  on  the  design 
of  distributed  computer  systems,  providing  a  quantitative  foundation  for 
decisions  that  vitally  affect  the  performance  and  cost  of  the  systems. 

The  stringent  constraints  associated  with  avionic  systems  make  such  systems 
a  fruitful  application  area  for  these  methods. 
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SECTION  II 

THE  SOFTWARE  ALLOCATION  PROBLEM 


2.1  INTRODUCTION 

We  are  interested  in  methods  for  the  optimal  allocation  of  a  set 
of  software  casks  to  a  sec  of  hardware  elements  consisting  of  pro¬ 
cessors,  memories  and  communication  links.  This  software  allocation 
problem  is  difficult;  it  is  a  dynamic,  stochastic,  highly  combinatorial 
problem  with  many  different  objectives. 

SCT's  approach  has  been  to  start  with  a  simple  model,  and  gradually 
increase  the  complexity  to  achieve  greater  realism.  In  the  previous 
contract  [17]  a  static,  deterministic  software  allocation  problem  was 
considered  in  which  a  known  set  of  tasks  with  known  requirements  and 
without  precedence  constraints  are  to  be  allocated  to  a  set  of  fully 
connected  processors  with  dedicated  memories.  The  objective  is  to 
minimize  the  completion  time  of  all  the  tasks.  There  are  also  constraints 
on  the  allocation;  each  task  can  only  run  on  a  subset  of  the  processors, 
some  tasks  may  have  bounds  on  their  finishing  times,  and  memory  con¬ 
straints  must  be  met. 

The  solution  technique  used  for  this  problem  has  been  spatial  dynamic 
programming  (SDP).  Tasks  and  processors  are  considered  as  nodes  of  an 
allowable  assignment  graph.  Nodes  are  processed  one  at  a  time,  to 
minimize  the  completion  time  of  the  processor  nodes  considered  thus  far. 

The  code  for  the  SDP  algorithm  appears  as  Appendix  A  of  this  report. 
It  is  written  in  PASCAL  for  a  DEC  VAX  11/780  and  consists  of  7  modules, 
which  are  then  linked  in  order  to  execute  the  program.  The  heart  of  the 
code  is  procedure  FIND  COST,  which  computes  the  maximum  processor 
finishing  time.  After  the  code  listing  a  sample  set  of  input  and  output 
appears  for  an  example  involving  8  processors  and  20  tasks. 
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The  amount  of  time  required  to  solve  a  software  allocation  problem 
depends  critically  upon  the  complexity  of  the  task-to-processor  assign¬ 
ment  graph.  Given  the  assignment  graph,  there  is  the  subproblem  of 
finding  the  optimal  order  to  process  the  nodes  in  the  SDP  method. 
Appendix  B  of  [17]  discusses  this  subproblem  in  the  case  where  each 
of  n  tasks  can  run  on  each  of  m  processors.  The  problem  complexity 
is  then  approximately  . 


This  is  an  upper  bound  on  the  complexity,  because  often  there  wil] 
be  sparsity  in  the  assignment  graph.  Using  the  notation  of  [39],  the 
complexity  is  approximately  equal  to 
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Now  it  is  still  difficult  to  tell  what 
all  that  is  known  is  that  |E^|  +  j^J  d^, 
of  the  kth  node.  Suppose  each  task  can  be  allocated  to  c  processors. 
Then  some  processor  must  be  able  to  handle  cn/m  tasks.  As  a  result, 
max(|E^|  +  |$k|)  >  cn/m.  Thus  expression  (2)  becomes  approximately 

(m  +  n)2cn/m  (3) 

assuming  that  n  >  m.  Expression  (3)  shows  qualitatively  what  has  been 
experienced  with  the  SDP  algorithm  in  practice.  If  the  number  of  tasks 
and  processors  both  increase  without  changing  their  ratio,  then  the 
amount  of  work  increases  linearly.  However,  if  the  number  of  tasks  alone 
increases,  or  the  graph  becomes  denser  (c  increases),  then  the  amount 
of  work  increases  exponentially. 
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The  software  allocation  problem  for  avionic  systems  is  dynamic 
and  stochastic.  Certain  tasks  must  execute  before  others.  Several 
methods  for  extending  the  SDP  algorithm  to  cover  the  allocation  problem 
with  precedence  relations  are  discussed  in  Section  2.2.  In  addition, 
there  may  be  uncertainty  about  exactly  which  tasks  must  execute,  or  how 
long  they  will  take.  Section  2.3  examines  a  method  for  analyzing  the 
behavior  of  processing  demands  in  a  stochastic  environment,  using  the 
methods  of  stochastic  network  theory.  These  methods  of  analysis  can  be 
used  to  test  the  effectiveness  of  a  software  allocation  algorithm. 

The  last  section  (2.4)  discusses  two  modifications  of  the  Dasic 
SDP  algorithm  to  correspond  to  a  more  realistic  software /hardware  model. 
The  first  modification  concerns  the  allocation  of  files  as  well  as  tasks. 
The  other  modification  allows  the  possibility  of  processor  failures. 

Tasks  can  be  allocated  to  more  than  one  processor,  with  the  more  important 
tasks  being  the  ones  most  likely  to  have  redundant  assignments. 

2.2  SOFTWARE  ALLOCATION  WITH  PRECEDENCE  CONSTRAINTS 

The  tasks  for  execution  in  real  avionic  software  systems  usually 
have  precedence  cgnstraints.  That  is,  certain  tasks  must  finish  executing 
before  others  are  allowed  to  start.  The  principal  reason  for  such  con*" 
straints  is  that  data  from  earlier  tasks  is  needed  for  those  coming  later. 

A  solution  for  the  allocation  problem  with  precedence  constraints 
requires  not  only  an  allocation  of  the  tasks  to  the  processors,  but  also 
a  scheduling  of  the  tasks.  The  time  that  each  task  begins  executing 
must  be  specified  and  when  a  task  starts,  all  tasks  which  come  before  it 
(as  dictated  by  the  precedence  constraints)  must  be  finished. 

Figure  2.1  shows  an  example  of  precedence  relationships  among  a 
set  of  20  tasks.  (This  example  is  taken  from  [17]).  A  feasible  schedule 
for  these  task3  on  a  complete  network  of  six  processors  is  shown  in 
Figure  2.2.  Notice  that  some  processors  have  "dead  time,"  in  which  no 
task  is  being  executed. 
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Figure  2.2.  Schedule  for  Example 
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The  precedence  problem  is  fundamentally  computationally  intensive. 

Thus,  any  solution  method  will  either  grossly  simplify  the  problem,  or 
require  an  excessive  amount  of  computation,  or  use  heuristics  to  arrive 
at  a  solution  which  is  most  likely  sub-optimal.  As  a  result,  it  seems 
advisable  to  discuss  several  different  approaches  to  this  problem,  each 
of  which  has  its  own  advantages  and  drawbacks.  However,  we  will  recommend 
one  method  (spatial  dynamic  programming  with  the  "method  of  windows")  which 
we  believe  is  the  best  approach. 

The  classical  methods  for  solving  such  problems  can  be  found  in  the 
area  of  operations  research  known  as  scheduling  theory.  Generally  speaking, 
a  set  of  tasks  is  to  be  scheduled  on  one  or  more  machines,  in  order  to 
optimize  some  performance  criterion.  Unfortunately,  all  but  the  simplest 
formulations  fall  into  a  class  of  problems  known  as  NP-complete,  which 
are  notorious  for  their  computational  intractability.  For  example, 

Hu  [32]  showed  that  if  there  are  m  identical  processors,  n  tasks  which 
each  require  one  time  unit,  and  if  the  precedence  constraints  form  a  tree, 
then  there  is  a  simple  algorithm  to  minimize  the  completion  time  of  all 
the  tasks.  If  the  tasks  are  not  identical  in  time  requirements,  or  if 
the  precedence  constraints  do  not  form  a  tree,  then  the  problem  is 
NP-complete  [38].  Lenstra  and  Rinooy  Kan  [38]  discuss  several  solvable 
precedence  problems  and  many  more  unsolvable  ones  (at  least  in  a  reasonable 
amount  of  time) . 

Only  a  few  of  the  papers  on  software  allocation  discuss  precedence 
constraints.  One  such  paper  [8]  is  the  most  recent  in  a  series  of  arti¬ 
cles  by  Kokhari.  In  [  8]  the  precedence  relationships  must  again  be  in 
the  form  of  a  tree.  Each  task  is  allocated  to  a  processor  and  to  a  time 
phase.  If  task  i  must  preceed  task  j  and  task  i  is  allocated  to  a  time 
phase  k,  then  task  j  must  be  allocated  to  the  same  phase  or  later.  The 
solution  method,  based  on  dynamic  programming,  finds  the  optimal  solution 
subject  to  the  predetermined  phases.  In  this  case  "optimal"  has  a 
different  meaning  -  the  sum  of  execution  and  communication  times.  As  a 
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result,  the  solutions  (such  as  the  example  in  [8])  are  likely  to  have  a 
significant  amount  of  dead  time,  which  may  not  be  acceptable.  Another 
drawback  to  this  method  is  the  difficulty  of  incorporating  memory  con¬ 
straints.  Thus  while  the  method  may  be  interesting,  it  is  unlikely  to 
solve  our  problem. 

Dr.  Ma  and  others  at  TRW  used  the  branch-and-bound  method  of  integer 
programming  to  solve  an  allocation  problem  in  [40].  The  method  involved 
a  form  of  precedence  constraint  in  that  several  "threads"  of  tasks  had 
to  satisfy  "port-to-port"  time  constraints.  A  "thread"  consists  of  a 
set  of  tasks  which  must  execute  consecutively.  However,  the  method  for 
determining  whether  these  time  constraints  are  met  is  not  explained. 

In  addition,  the  optimization  criterion  of  minimizing  interprocessor  com¬ 
munication  may  result  in  unbalanced  processor  loads. 

Instead  of  trying  to  find  an  optimal  solution  to  a  simple  formulation, 
a  suboptimal  solution  to  a  complex  formulation  may  be  found  via  heuristics. 
One  heuristic  method  for  the  allocation  problem  with  precedence  constraints 
is  described  in  the  final  report  of  this  contract's  predecessor  [17].  It 
is  based  on  the  longest-path  method  used  by  Kaufman  [34].  The  modifications 
described  in  our  report  incorporate  inter-task  communication  and  task 
memory  requirements.  Several  examples  are  given  there  showing  how  the 
method  is  used.  The  primary  drawback  to  this  method  is  that  because  it 
is  heuristic  it  is  unlikely  to  produce  optimal  solutions.  In  many  cases 
the  solutions  will  be  near-optimal,  but  sometimes  they  may  be  far  from 
optimal.  However,  for  problems  which  are  large  enough  (say,  15-20  tasks 
per  processor)  that  more  complex  methods  (such  as  the  "method  of  windows" 
described  below)  are  computationally  prohibitive,  a  good  heuristic  method 
such  as  this  is  essentially  the  only  way  to  produce  an  acceptable  allo¬ 
cation. 

The  method  we  recommend  for  solving  the  allocation  problem  with 
precedence  constraints  is  spatial  dynamic  programming,  with  certain  modifi¬ 
cations.  The  primary  reason  for  this  is  that  the  SDP  method  is  very 
general.  Several  different  performance  criteria  and  many  types  of  con¬ 
straints  can  be  incorporated. 
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The  modifications  are  called  the  "method  of  windows."  For  each  task 
we  set  up  a  "window"  of  time  in  which  it  may  execute.  The  window  is 
larger  than  the  task's  execution  time.  If  task  A  must  precede  task  B, 

A's  window  should  come  before  B's.  Any  allocation  which  satisfies  the 
window  constraints  automatically  satisfies  the  precedence  constraints. 

Figure  2.3  shows  an  example  of  a  precedence  tree.  Assume  that  this 
is  one  of  several  in  a  particular  allocation  problem.  The  execution  times 
are  given  in  the  second  column  of  Table  2.1.  Next  the  windows  need  to  be 
established.  The  last  column  of  Table  2.1  gives  one  set  of  windows 
which  forces  the  tasks  to  satisfy  the  precedence  constraints. 

Finding  an  appropriate  size  for  the  windows  must  be  done  on  a  heuristic 
basis.  Factors  which  should  be  taken  into  account  include  the  number  of 
processors  and  the  amount  of  parallelism  in  the  software  structure.  If 
the  windows  are  too  small  there  will  be  no  feasible  solution,  and  the 
windows  should  be  enlarged.  If  the  windows  are  too  big  there  will  be  a 
significant  amount  of  dead  time  in  the  schedule,  making  the  completion 
time  larger  than  it  should  be.  This  can  be  remedied  by  either  compressing 
the  schedule  to  remove  some  dead  time  or  by  shrinking  the  window  sizes. 

Once  the  windows  are  established,  the  SDP  algorithm  proceeds  as  in 
the  static  case  with  no  precedence  relationships.  However,  now  there 
are  additional  constraints  which  must  be  checked  at  each  processor  node. 

For  each  set  of  tasks  which  may  be  allocated  to  the  processor,  it  must 
be  determined  whether  there  is  a  feasible  solution  which  meets  the 
window  constraints.  If  there  are  k  tasks  in  the  candidate  allocation, 
there  are  k!  possible  execution  orders.  Conceivably  each  of  these 
may  have  to  be  examined  to  see  if  there  is  one  which  meets  the  con¬ 
straints. 

As  an  example,  consider  the  situation  shown  in  Figure  2.4.  Only 
one  order  of  execution,  A  then  C  then  B,  will  satisfy  the  window  con¬ 
straints.  However,  if  B  required  more  than  15  time  units  there  would 
be  no  feasible  schedule. 
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a 


Task 

Execution  Time 

Window 

A 

10 

0-15 

8 

20 

15  -45 

C 

10 

45-65 

D 

15 

45-70 

E 

25 

70-110 

F 

5 

Table  2.1 

Execution  Times  and  Corresponding  Windows 


Extcucion 


Figure  2.4.  Example  of  Task  Windows 
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If  k  is  not  more  than,  say  10,  Lt  is  feasible  to  consider  all  p,  ssible 
orders.  McMahon  and  Florian  [41]  give  an  efficient  method  of  enumerating 
all  possibilities.  For  larger  values  of  k,  heuristic  methods  may  have  to 
be  used.  Baker  and  Su  [4]  showed  that  some  heuristics  can  often  determine 
whether  a  feasible  schedule  exists.  If  the  heuristics  do  not  come  up  with 
a  feasible  solution,  the  windows  may  have  to  be  enlarged. 

As  an  example  of  how  the  method  of  windows  works,  consider  the  two 
software  functions  shown  in  Figure  2.5.  (Admittedly  this  is  a  small  example, 
but  it  will  illustrate  the  method).  The  data  for  the  tasks  is  shown  in 
Table  2.2.  The  windows  were  chosen  to  be  about  50 %  larger  than  the  execution 
times,  and  to  preserve  the  precedence  constraints.  The  hardware  will  be 
three  fully  connected  processors,  each  with  a  dedicated  memory  with  capacity 
40.  Figure  2.6  shows  the  network  of  possible  assignments. 


Figure  2.5.  Two  Software  Functions  with  Precedence  Constraints 
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EXECUTION 

TIME 

- — - 

WINDOW 

MEMORY 

REQUIREMENT 

10 

0-15 

20 

5 

15-25 

10 

15 

25-50 

20 

10 

0-30 

25 

20 

0-30 

10 

15 

30-50 

10 

Table  2.2 
Data  for  Example 


Figure  2.6.  Network  of  Possible  Assignments 
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One  point  to  notice  is  that  the  window  constraints  are  binding.  For 
example,  there  is  sufficient  memory  for  tasks  A,  B,  and  E  all  to  be  located 
at  processor  2.  However,  not  all  of  the  task  window  constraints  can  be 
met . 

Using  SDP,  an  optimal  allocation  given  these  windows  is  found  to  be 
assigning  task  4  to  processor  1,  tasks  1,  2,  and  6  to  processor  2,  and 
tasks  3  and  5  to  processor  3.  The  resulting  schedule  is  shown  in  Figure  2.7. 
As  expected,  there  is  a  significant  amount  of  dead  time.  The  schedule  can 
be  compressed,  being  careful  to  maintain  the  precedence  constraints,  as 
shown  in  Figure  2.8. 


Figure  2.7.  Schedule  Produced  by  SDP  Algorithm  and  "windows" 
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Figure  2.8.  Compressed  Schedule 

We  have  examined  several  methods  for  attacking  the  software  allocation 
problem  when  there  are  precedence  constraints.  Another  important  feature 
in  avionic  systems  is  that  the  casks  needed  on  a  mission  and  their  execution 
times  are  uncertain.  The  next  section  discusses  a  method  for  evaluating 
allocations  in  a  stochastic  environment. 

2.3  STOCHASTIC  VARIABILITY  AND  SOFTWARE  ALLOCATION 

We  are  concerned  in  this  section  with  a  network  of  processors  and 
associated  software  functions  for  which  the  input  of  jobs  (requests  for 
execution  of  software  functions)  is  uncontrolled,  movements  (routes)  of 
jobs  throu?*  the  system  and  class  changes  of  jobs  are  random,  resources 
required  to  process  jobs  vary  randomly,  time  horizons  of  shorter  duration 
than  necessary  to  achieve  "steady-state"  are  of  interest,  and  where  it  is 
important  to  respond  to  demands  with  minimal  competition  for  available 
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processing  resources.  We  want  to  account  for  these  various  factors  through 
an  appropriate  network  model  and  provide  useful  and  computationally  tractable 
results  for  performance  prediction.  The  model  structure  we  consider  is 
reminiscent  of  Baskett  et  al  [7]  but  the  approach  taken  in  analyzing  the 
model  is  prompted  by  Harrison  and  Lemoine  [29]. 

In  this  model,  a  "job"  corresponds  to  a  sequence  of  tasks.  As  the  job 
moves  through  the  network  of  processors,  different  tasks  are  being  executed. 
Changing  the  "class"  of  a  job  corresponds  to  moving  from  one  task  to  another 
within  a  job.  Admittedly  this  is  different  terminology  than  has  appeared 
previously;  it  is  used  to  correspond  to  the  terminology  in  the  stochastic 
network  theory  literature. 


This  section  is  organized  as  follows.  The  basic  features  of  the 
model  are  described  in  2.3.1.  Following  this,  we  consider  in  2.3.2  the 
movement  of  a  typical  job  through  the  network  with  particular  emphasis 
on  the  resources  required,  global  as  well  as  nodal,  to  process  the  job 
until  it  leaves  the  system.  We  then  consider  in  2.3.3  the  evolution  of 
the  network  over  time  when  the  pattern  of  demand  input  is  a  Poisson 
process  and  the  processing  resources  available  at  each  node  in  the  network 
are  effectively  unlimited  (e.g.,  there  are  sufficient  servers  at  each 
node  so  that  objects  never  wait  in  queue).  An  explicit  representation 
is  given  for  the  time-dependent  distribution  of  network  state  (defined 
here  as  the  numbers  of  jobs  of  various  types  at  each  node  in  the  system). 

When  the  Poisson  input  process  is  homogeneous,  this  time-dependent  dis¬ 
tribution  converges  to  a  limit,  and  the  "distance"  between  the  time-dependent 
and  limiting  distributions  is  estimated.  Some  implications  of  the  results 
for  design  and  performance  analysis  are  indicated.  Indeed,  the  basic 
notion  behind  the  approach  of  2.3.3  is  to  first  determine  a  (time-dependent) 
distribution  on  the  level  of  resource  usage  in  the  system  assuming  ample 
resources  are  available,  and  to  then  use  this  distribution  for  identifying 
candidate  resource  capacity  design  parameters  and  assessing  "adequacy"  via 
confidence  intervals  obtained  from  the  distribution. 
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2.3.1  Model  Description 

Consider  a  network  in  which  jobs  in  different  classes  are  moving 
through  the  system,  and  also  possibly  changing  class,  in  a  possibly  random 
manner,  and  requiring  a  possibly  random  amount  of  resource  (program,  pro¬ 
cessing,  memory,  time,  etc.)  at  each  node  visited.  There  are  N  nodes 
(processing  centers,  clusters,  etc.)  indexed  by  i  and  M  job  classes 
indexed  by  m.  Inputs  for  processing  at  any  node  may  originate  directly 
from  outside  the  system  or  by  internal  transfer  within  the  network.  On 
any  visit  to  node  i  by  a  job  in  class  m  the  job  is  completed  (routed 
to  an  artificial  sink  in  the  network  after  processing  at  i  with  probability 
q^,  independently  of  previous  visits,  classes  and  other  jobs  present  in 
the  system;  and,  in  this  event,  the  amount  of  resource  required  at  node  i 

to  process  the  job  is  distributed  as  a  random  variable  S  . .  Further, 

mi 

on  any  visit  to  node  i  by  a  job  in  class  m  the  job  remains  in  the  system 
after  processing  at  node  i  and  transfers  to  node  j  as  a  job  in  class  r 
with  probability  p*j”j,  independently  of  prior  visits,  classes  and  other  jobs 
(if  i=j  then  the  job  is  sent  back  to  node  i) ,  where 


'mi 


N 

I 


M 

z 

>1  r=l 


mr 

Pij 


for  each  m  and  i;  and,  in  this  event,  the  amount  of  resource  required 
at  node  i  to  process  the  job  is  distributed  as  a  random  variable 
Processing  requirements  for  all  visits  by  all  jobs  in  all  classes  to  any 
node  are  independent,  and  all  jobs  eventually  exit  from  the  system.  The 
model  dynamics  can  be  summarized  by  two  examples.  The  top  of  Figure  2.9 
shows  two  software  functions  A  and  B  with  separate  external  input. 

In  function  A,  there  is  a  stochastic  transition  after  task  1:  task  2 
is  executed  with  probability  p  and  task  3  is  executed  with  probability 
1-p.  In  function  B,  after  task  7  finishes  the  function  is  again  executed 
with  probability  1-q.  This  might  correspond  to  a  tracking  function, 
where  the  tracking  stops  if  the  object  being  tracked  leaves  the  radar 
screen. 
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The  bottom  of  Figure  2.9  shows  a  possible  allocation  of  these  tasks  to  a 
network  of  four  processors.  The  transitions  which  are  actually  made  depend 
on  the  particular  task  being  executed  at  a  processor.  For  example,  from 
the  processor  where  tasks  1  and  5  are  allocated  execution  proceeds  to  the 
right  if  either  task  2  or  task  6  follows,  while  it  goes  to  the  processor 
below  if  task  3  follows.  All  execution  streams  eventually  enter  the  sink 
with  probability  1. 

The  preceding  formulation  can  easily  accommodate  multiple  resource 
types  by  specifying,  for  each  type  of  resource,  appropriate  probability 
distributions  for  Rmr  and  S  .  for  m,r=l,...,M  and  i,j*l,...,N. 

ij  mi 

In  what  follows  we  do  not  distinguish  between  multiple  resource  types  in 
order  to  keep  the  notation  from  being  excessively  complicated,  but  the 
results  presented  on  resource  requirements  are  valid  for  each  resource  type. 

2.3.2  Routing  and  Resource  Requirements 

Consider  now  the  route  and  class  of  a  generic  job  moving  through  the 

network.  Let  b  be  the  probability  that  a  job  enters  the  svstem  through 
ml 

node  i  in  class  m.  For  i,j=l,...N  let  P„  =  [p^  ]  where  m,r=l,...,M 
so  that  is  a  tl  by  M  matrix,  and  then  put  P  =  [P  ]  so  that  P 

is  a  N  by  N  block  matrix.  Let  I  be  the  NM  by  NM  identity  matrix. 
The  routing- and-class  history  of  a  typical  job  moving  through  the  system 
corresponds  to  the  evolution  of  a  finite  Markov  chain  with  a  single  ab¬ 
sorbing  state  (namely,  the  network  sink)  and  non-absorbing  states  {  (m,i): 
m=l,...,M  and  i=l,...,N},  with  transitions  among  non-absorbing  states 
governed  by  P.  Since  all  jobs  eventually  exit  the  system  (i.e.,  the 
chain  is  absorbing)  the  matrix  I  -  P  is  invertible,  and 

OO 

[I  -  Pf1  =  l  PX  . 

x=0 

If  we  put  (I  -  P)  1  =  [A.,1  and  A,,  =  [amV,]  then  a'ff  is  the  expected 

ij  ij  ij  ij 

number  of  requests  for  processing  at  node  j  by  a  job  in  class  r  which 
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originally  enters  the  network  through  node  i  in  class 
the  probability  a  job  entering  via  node  i  in  class  m 
in  class  r  at  some  later  step.  Then 

TIT- 

otherwise.  Moreover,  if 
network  via  node  i  in  class  m 

and 


mm  ,  ,  mm,  -1 

Cii  =  1  -  (aii) 


m.  Let  c.  .  be 
reaches  node  j 

TT1T 

and 


mr  mr  ,  rr 
..  .  =  a.  .  /a.  . 
ij  ij  31 


q^  ^  is  the  probability  that  a  job  entering  the 


exits  the  system  from  node  j 


,  rr 

then  q. 

11 


q  ./( 1  -  Crr) 

r]  11 


mr 

qii 


in  class  r 

mr  rr  ,  .  _ .  . ,  * 

.  q_  otherwise.  Finally,  if  q 


il  11  •mi 

is  the  unconditional  probability  that  a  job  exits  the  system  from  node  i 


in  class  m  then 


„  ^  —  ,  L  ill 

Q,.  =  q  .  I  i  b  .  a.  .  . 

ml  1111  •  ,  ,  ri  1 1 

j=l  r=l  J  J 

mr  mr  * 

The  above  expressions  for  the  probabilities  p.  ,  q.  and  q  follow 

ij  ij  mi 

immediately  from  well  known  results  for  finite  absorbing  Markov  chains; 
cf.  Kemeny  and  Snell  [35].  Observe  that  the  crucial  factor  in  determining 
these  various  routing  and  class  parameters  is  computation  of  the  matrix 
(I  -  P)  1.  In  many  applications  of  interest  this  matrix  I  -  P  will  be 
relatively  sparse,  indeed  even  upper  triangular. 


Turning  to  global  network  resource  requirements,  let  L  .  be  the 

ml 

total  level  of  resources  necessary  to  process  a  job  until  it  reaches  the 

network  sink  given  that  the  job  enters  the  system  at  node  i  in  class  m. 

For  0  >  0  let  f  .(9),  g™5(9),  and  h  .(9)  be  the  Laplace  transforms 
—  mi  ij  mi 

for  the  distributions  of  S  .,  R.  ,  and  L  respectively.  Bv  virtue 

mi  ij  mi 

of  the  independence  assumptions  regarding  routing  and  class  and  resource 

requirements  on  visits  to  nodes,  the  transforms  {h  .(9):  m  =  1,...,M  and 

nil 

i  =  1,...,N}  satisfy  the  network  flow  equations 


h  .(8)  =  q  .f  .(0) 

mi  mi  mi 


N  M 
Z  Z 
j=l  r=l 


mr  mr.,. 
PU  Si3(r) 


hrj(6) 


(1) 


for  m=l,...,M  and  i=l,...,N.  This  system  of  equations  has  a  unique 
solution  as  follows:  Let  f ( 6 >  and  h ( © )  be  NM  column  vectors  con¬ 
sisting  of  N  strings  of  length  M  where  entry  m  of  string  i  in 

f(9)  is  q  .f  .(9)  and  entry  r  of  string  j  in  h(9)  is  h  .(9). 

mi  mi  rj 
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Lac  DC1?)  be  a  N  by  N  block  matrix  where  the  (ij)th  block,  say 
is  the  M  by  M  matrix  [ p™ J  g”^(0)].  Then  (1)  is  equivalent  to 


V^’ 


h(0)  =  f(e)  +  D(9)  h(9) 


(2) 


or 


[I  -  D(6)]  h(0)  =  f(0). 


Since  0  <_  D(0)  <_  P  component -wise  it  follows  that  the  matrix  [I  -  D(9)] 
is  invertible  for  any  9  >  0;  indeed. 


[I  -  D(0) ] 


I 

x=0 


[D(9)]X  . 


Thus 


h(9)  =  [I  -  DO)]"1  f  (9) .  (3) 

Computation  of  the  transforms  {h  .(6)}  from  (3)  is  no  simple  matter,  but 

mi 

we  will  provide  an  explicit  representation  for  these  transforms  using  a 
slightly  different  approach. 

On  the  other  hand,  repeated  differentiation  of  the  network  flow 

equations  with  respect  to  9,  and  then  setting  9=0,  provides  a  recursive 

procedure  for  computing  the  moments  (first,  second,  third,  and  so  on)  of  the 

cumulative  resource  variables  (L  For  n  =  1,2,...  let  h  . (n)  =  E{  (L  .  )n}, 

mi  mi  mi 

(n)  =  E{ (Rmr)n)  and  f  . (n)  =  E{  (S  ,)n}.  We  assume  henceforth  that 
ij  ml  ml 

and  S  have  finite  moments  of  all  orders  for  m,r=l,...,M  and 
ml 

i,j=l,...,N;  the  distributions  customarily  used  in  the  modeling  of  service 

systems  have  this  property,  and  so  the  assumption  is  not  really  restrictive. 

We  will  now  observe  that  the  variables  {L  .}  also  have  finite  moments 

mi 

of  all  orders  and  provide  a  recursive  procedure  for  computing  these  moments. 


mr 

3ii 

,mt 

ij 
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Differentiating  both  sides  of  (1)  n  times  with  respect  to  -  and 
using  Leibnitz's  Rule  leads  to 

h(n)(9)  =  f(n)(6)  +  Z  (  £  ]  D(x)  (6)h(n-x)  (0)  , 


hCn)(9)  =  [I  -  D(9)P]_1  ^f(n)(6)  +  Z  D(x)  (9)h(n-x)  (8)^  (4) 


where  the  superscripts  correspond  to  component-wise  differentiation  in  (2)  and 

-Jll _  • 

\xl  x.'(n-x)! 

Suppose  that  each  of  the  variables  [  (L  .  )n  1}  has  finite  mean  (this  cer- 

tainly  holds  for  n  =  1) .  Then  each  component  of  ti  n  x'(9)  has  a  finite 

limit  as  9-0  for  x=l,...,n.  Since  [I  -  D(9) ]_1  -  [I  -  p]"1  as 

-  "  0,  it  now  follows  from  (4)  that  each  component  of  h1"11^  (5)  has  a 

finite  limit  as  6  0,  and  so  each  of  the  variables  {  (L  .)n’-  las  finite 

mi 

mean.  Thus,  we  conclude  by  mathematical  induction  that  each  of  the  variables 
'  Lmi^  has  moments  of  all  orders.  Moreover,  using  (4)  we  can  give 

a  compact  recursive  formula  for  computing  these  moments.  For  x=l,2,... 
let  <5^,  h^  and  h^  be  NM  column  vectors  where  the  entries  corresponding 
to  (m,i)  are  q^f^,  h^,  and  1,  respectively;  also,  let  be  a  NM 

by  NM  matrix  where  the  entry  corresponding  to  (m,i),  (r,j)  is  pmrgmT(x) . 
Then  1J  1J 


h  =  [I  -  P]"1  1*5  +  Z  (  n  )  D  h 

n  n  .  '  x  /  x  n-x 

L  x=l  J 


for  n=l,2,...  .  The  notable  feature  of  the  recursion  formula  (5)  is  that 

it  requires  only  the  given  network,  parameters  (i.e.,  the  matrix  P  and 

the  moments  of  the  variables  ‘  Rmf,  S  . :)  and  computation  of  the  matrix 
-1  ij  mi 

[I  -  P]  \ 


With  regard  to  nodal  (or  local)  resource  requirements,  let  Lmr  be  the 
,  ij 

total  level  of  resources  required  at  node  j  to  process  a  job  in  class  r 

which  enters  the  network  at  node  i  in  class  m.  Put  another  wav,  Lmr 

ij 
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is  che  total  loading  induced  at  node  j  by  an  object  in  class  r  entering 
at  node  i  in  class  m.  Observe  that 

N  M 

L  a  £  v  Lffir 

mi  .  ,  ij 

j»l  r»l  J 

Let  4)^(0)  be  Che  Laplace  transform  for  the  distribution  of  L?r.  For  an 

^  ss  ^ 

arbitrary  node  k  and  job  class  s  let  q  +  uslt  =  1  ~  and 


W9>  = 


Wsk(9)  +  uskgsk(9) 

,  ss  sk,A. 

1  * 


The  quantity  u  ^  is  the  probability  that  a  job  at  node  k  in  class  s 
departs  the  network  either  without  returning  to  node  k  or  from  node  k 
but  in  a  different  class.  We  then  have 


<j>4(e) 


^(9) 


if  i=j  and  m=r,  and 


I  ,  mr  .  mr 
'  1  -  p,  +  p  < 


>.  +  p  .4)  .(6)  otherwise  . 

ij  ij  r j 


Moreover,  in  reference  to  the  comment  following  (3)  above, 

■  n  n 

j=l  m=l  J 

The  transforms  {4>™j(9)}  as  given  above  make  possible  computation  of  the 
moments  of  the  variables  { L™p .  For  example: 


E{L^} 


q  , f m,(l)  +  (1-q  .)g“T(l) 
mi  nu.  _ mi  mi 

.  mm 

1  -  cu 


E{  a-, 2, .  yf.i(2>  1  (1-v>02>  1 9  , 

ii  ,  mm 

1  -  Pii 
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E{Lmr}  = 
1J 


pmr  E{  l™0} 
ij  ii 


E{  (Lmr)2}  = 


mr 

Pij 


E{  (L™)2} 


Finally,  let  L  be  the  total  resources  required  at  node  i  in 
mi 

class  m  by  a  typical  job  entering  the  network..  Then  the  distribution 

■k 

of  has  Laplace  transform 


N 

l 

3-1 


M 

I 

r=l 


b  ,frm(9) 
rj  Ji 


2.3.3  Poisson  Input 


We  consider  now  the  evolution  of  the  network  over  time  when  jobs  enter 

the  system  according  to  a  Poisson  process  A  =  {A(t),  t  _>  0}  with  intensity 

function  { A ( t) ,  t  >  0}  and  when  the  processing  resources  available  to  each 

node  in  the  network  are  effectively  unlimited  (e.g.,  in  queueing  parlance, 

there  are  sufficient  servers  available  at  each  node  so  that  jobs  never 

wait  in  queue) .  The  variables  Rmr  and  S  .  are  then  interpreted  as 

ij  mi 

processing  times  at  node  i.  The  discussion  we  give  follows  Section  3 
in  Harrison  and  Lemoine  [29]. 


The  assumption  of  Poisson  input  as  stated  above  means  that  the  numbers 
of  jobs  arriving  in  non-overlapping  time  intervals  are  independent  random 
variables  and  that  for  n  *  0,1,2,... 


P{  A(t)  =  n; 


e~A(t) 


[A(t)]n 

_  I  > 


t  >  0, 


where 


t 

A(t)  =  J  \(y)dy  . 
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In  particular,  if  A  .  (t) 
ml 


is  the  number  of  jobs  entering  the  network  through 


node  i  in  class  m  up  to  time  t  and  A 


mi 


{A  (t) ,  t  >  0} ,  then 
mi  — 


{A  m=l,...,M  and  i=l,...,N}  are  independent  Poisson  processes  and 
ml 

A  .  has  intensity  function  {b  A(t),  t  >  0}  . 
mi  mi  — 


The  histories  (routes  and  classes)  of  jobs  moving  through  the  system 
are  independent  and  distributed  as  a  process  Y  *  (Y(t),  0  <_  t  <_  L;  where 
Y(t)  e{(m,i):  m=l,...,M  and  i=l,...,N}  with  the  variable  L  interpreted 
as  the  total  length  of  time  a  generic  job  is  in  the  network.  In  particular, 

L  has  the  same  distri- 


Y(0)  *  (m,i)  with  probability  b  . ,  in  which  case 

nn 


bution  as  the  variable  L 


mi 


Now  define 


Yr,(t)  =  P(L  >  t,  Y(t)  =  (r ,  j ) } 


N  M 


=  Z  E  b  ,«L  ^  c.  Y(t)  “  (r.j)  Y(0)  -  (m.i)}. 

.  i  ,  mi  mi 
i=l  m=l 


5rJU)  H  /  x(y>Yrj(t-y)dy, 


and 

N  M  t 

C(t)  =  l  Z  Z,  (t)  =  /  X(y)P{L  >  t-y } dy  . 
j=l  r=l  J  o 

Assuming  the  network  is  empty  at  time  0,  the  quantity  ^^(t)  is  the 
expected  number  of  jobs  occupying  node  j  in  class  r  at  time  t  and 
£(t)  is  the  expected  number  of  jobs  in  the  system. 


Now  let  Cr_.(t)  be  the  number  of  jobs  occupying  node  j  in  class 
r  at  time  t  and  let 


<j?(t)  ={Crj(t):  r=l,...,M  and  j  =  l,...,N} 
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be  the  "state"  of  the  network  at  time  t.  Also,  let  C  *  {c  }  be  a 

rj 

generic  state  of  the  system.  Then  the  following  remarkable  result  holds 
(cf.  [29]): 

N  M 

K^(t)  =  C}  =  e~C(t)  f]  n  Ur.(t)]Crj/c  .  (6) 

j-1  r=l  rj 

That  is,  the  number  of  jobs  occupying  node  j  in  class  r  at  time  t 
has  a  Poisson  distribution  with  mean  £  (t) ,  and  the  numbers  occupying 

the  various  nodes  and  classes  are  independent  random  variables. 


Implementation  of  (6)  requires  computation  of  the  mean  values  { 
which  is  difficult  for  non-homogeneous  input.  Suppose,  however,  that 
a  homogeneous  Poisson  process  with  A(t)  =  A  for  t  >  0.  Let 


(t)} 

is 


and 


N  M 
T  Z 

i*l  m=l 


V*1- 


mr. 


N 

5  =  E 
j-1 


M 

E 

r=*l 


r 

"rj 


In  the  homogeneous  case  we  have 
(6)  that 


r: 


as  t*00. 


It  then  follows  from 


N  M 

lim  P«f(t)  *  C)  =  e“"  [I  FI  (5  .)  rj/ 

t-«  j-1  r=l  c 


(7) 


This  limiting  or  asymptotic  distribution  depends  only  on  the  arrival  rate  A 
and  the  expected  amount  of  time  that  a  job  spends  at  node  j  in 

class  r  while  in  the  network,  and  not  upon  the  forms  of  the  distributions 
for  processing  times  at  the  various  nodes.  Moreover,  the  numbers  of  jobs 
occupying  the  various  nodes  and  classes  are  independent,  Poisson  distributed 
ran dom  variables. 
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In  the  case  of  homogeneous  input,  the  time-dependent  distribution  of 
state  given  by  (6)  can  be  approximated  by  the  asymptotic  distribution  given 
in  (7)  when  t  is  large  enough.  The  closeness  of  this  approximation  can 
be  gauged  as  follows.  Let  ^(C)  denote  the  right  side  of  (6)  and  tt(C) 
the  right  side  of  (7)  and  then  define 

d(tr  ,tt)  =  E  (tt  (C)  -  tt(c)  j  . 

C 

We  can  think  of  d^^.TT)  as  the  distance  between  the  distribution  of  state 
at  time  t,  namely  tt  ,  and  the  asymptotic  distribution  of  state  it.  (In 
the  terminology  of  measure  theory,  d(tr  ,tt)  is  the  total  variation  of  the 
signed  measure  it— rr  .)  For  the  multidimensional  Poisson  distributions  it 
and  tt,  it  .-an  be  shown  (cf.  Cinlar  [13,  pp.  564-565])  that 

N  M 

d(\,ir)  1  2  £  Z  Kr.  ~  5  .(t )]  . 

3-1  r-1  r3  r3 

But  the  summation  on  the  right  side  of  this  inequality  is 


5  -  £(t)  =  \J  P{L  >  y)dy  . 

By  Chebyshev's  Inequality  (cf.  Chung  [12,  p.  48]),  we  have  P{  L  >  y}  <_  E(  Ln}/yn, 
whence 

CO 

/  P(L  >  y}dy  <_  E{  Ln)  f  y"ndy  =  E{  Ln)  /  (n-1)  tn_1 
t  J  t 

for  n“2,3,...,  where 


N  M 

E{  Ln}  *  E  E  b  .  E{  (L  ,)n} 
J-l  r-l  r3 


and  the  moments  of  the  variables 


using  (5).  Thus,  if 


can  be  computed  recursively 
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U  (t)  -  [X/ (n-l) tn_1]  E{ L°} 
n 


Chen 


£  -  Mn(t)  <_  £(t)  <_  E, 


and 


d(TT  T)  <  2  p  (t)  . 

t  —  n 

Similar  results  hold  for  the  marginal  distributions  of  it 
Indeed,  if  B  is  a  subset  of  {(r,j):  r=l,...,M  and  j=l,... 
tt  and  tt  the  corresponding  marginal  distributions  of  tt 

o  t  D  t 

respectively,  then 


Z  £ 

B 


rj 


^BnU) 


<  I  £  .(t)  < 
-  n  rj 


’rj 


and 


d<*Bt’V  i 


^Bn(t) 


whe  re 


U_(t)  =  [A/(n-l)tn_1]  Z  b  E{(L  )“}  . 

B  rj  rj 

Horeover,  if  X  has  a  Poisson  distribution  with  mean  S  then 
et  al  [30,  p.  107] 


P{X  <  0/2}  < 


and 


P{  X  >  28}  <  (eM)8  . 


(8) 

and  tt. 

N} ,  and 
and  tt. 


(cf.  Hoel 
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Letting 


v(c) 


1 

B 


it  then  follows  that 


P(crj(t)  <_  £  (t)/2  for  all  (r, j)  in  B} 


n  hc  (t)  <  $  (t)/2j 


rj  -  rj 


i  n  i^ri)"rj( } 

B 


(9) 


<  (/27I)aBn(C) 


and 


H  C  (t) 


2V° 


for  all  (r,j)  in  B}  <  (e/4) 


(10) 


Finally,  if  “S’  »  {C^}  is  a  random  vector  having  distribution  it,  then 

letting  t-*00  in  (9)  and  (10),  we  see  that 


P(C*  <_  5rj/2  for  all  (r,J)}  <_  (/2/e)? 

and 

PtC^  >  2Srj  for  all  (r,j)>  <_  (e/4)C  . 


(11) 


(12) 
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The  preceding  results  suggest  that,  if  possible,  the  resources  placed 

at  node  j  (1  <_  j  <  N)  should  be  adequate  to  simultaneously  process  at 

least  2£  .  jobs  in  class  r  (1  <  r  <  M) ,  whenever  the  external  input 

process  is  Poisson  with  rate  X.  More  generally,  if  a  fixed  amount  of 

processing  capacity,  say  P  ,  is  available  to  the  system  as  a  whole,  then 

o  M 

the  portion  of  that  capacity  placed  at  node  j  should  be  (E  5  . /S)P  • 

r=l 


2.4  EXTENSIONS  OF  SDP 


The  spatial  dynamic  programming  methodology  was  used  to  solve  a 
deterministic  software  allocation  problem.  In  this  section  we  consider 
two  extensions  of  the  methodology  in  order  to  achieve  a  more  realistic 
model.  First,  the  method  is  extended  to  allocate  files  as  well  as  tasks. 
Second,  allocation  with  the  possibility  of  hardware  failures  is  considered. 

Previously,  communication  was  assumed  to  occur  directly  between  tasks. 
In  reality,  when  a  task  finishes  executing  it  may  write  its  results  to  a 
file.  Another  task  reads  the  file,  then  uses  the  data  to  perform  its  job. 
The  file  must  be  stored  in  a  memory,  and  thus  requires  a  certain  amount  of 
space.  The  allocation  of  files  must  be  done  along  with  the  allocation  of 
tasks . 


The  extension  in  this  case  is  quite  simple.  We  consider  a  file  to  be 
a  "task"  which  requires  memory  space  but  has  no  instructions.  It  communi¬ 
cates  with  all  tasks  that  write  to  it  or  read  from  it.  A  certain  amount 
of  memory  is  needed  for  each  file.  We  can  then  allocate  the  files  along 
with  the  tasks  in  the  SDP  algorithm. 

The  second  extension,  involving  hardware  failures,  is  far  more  complex. 
For  clarity,  the  problem  will  be  simplified  to  that  of  minimizing  the 
finishing  time  of  all  tasks,  subject  to  memory  constraints.  Other  con¬ 
straints,  such  as  communication  and  task  execution  time,  can  easily  be 
incorporated. 
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We  will  assume  that  each  hardware  element  (processor  with  dedicated 
memory)  has  an  independent  probability  of  failure,  say  p.  This  is  a 
rather  strong  assumption,  but  it  greatly  reduces  the  amount  of  computa¬ 
tion  required.  Processor  dependent  failure  probabilities  can  be  accommo¬ 
dated,  with  a  consequent  increase  in  computational  load. 

Important  tasks  should  be  assigned  to  more  than  one  processor.  If  a 
processor  fails,  the  tasks  can  still  execute  on  another  processor  to  which 
they  have  been  allocated.  If  p  is  small,  it  is  unlikely  that  more  than 
one  processor  will  fail.  As  a  result,  we  will  assume  that  no  task  is 
assigned  to  more  than  two  processors.  If  a  task  is  assigned  to  two,  then 
one  is  designated  the  primary  assignment  and  the  other  the  secondary  assign¬ 
ment.  The  task  runs  on  the  primary  processor  unless  that  processor  fails, 
in  which  case  the  task  runs  on  the  secondary  processor. 

The  optimization  criterion  should  be  based  on  the  performance  of  the 
system.  If  there  are  no  hardware  failures,  we  can  again  minimize  the  task 
finishing  time.  Without  memory  constraints,  every  task  would  be  allocated 
to  two  processors  and  would  be  certain  of  executing.  However,  if  some 
tasks  are  allocated  to  only  one  processor,  these  may  not  execute.  This 
should  degrade  performance  somewhat. 

Define  d^  to  be  a  penalty  which  is  incurred  if  task  i  is  not 
executed.  This  value  can  be  high  for  critical  tasks,  low  for  less  important 
ones.  It  can  be  in  any  units  meaningful  to  the  decision-maker.  If  tasks 
i^,  ...,  i^  are  not  executed,  the  total  penalty  is  some  function 

f (d .  ,  d,  ,  ...,  d.  ). 

1  2  \ 

Now  we  need  to  combine  the  two  attributes  of  time  and  penalty.  For 

a  given  allocation  and  a  given  realization,  let  T^  be  the  completion 

time  of  processor  j  and  D  the  set  of  non-executed  tasks.  We  can  then 

postulate  a  multi-attribute  utility  function  U,  of  the  form  U(max  T  ,  f (D) ) 

1 

which  is  decreasing  in  both  arguments.  The  objective  is  to  find  J 
the  allocation  which  maximizes  the  expected  utility. 
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This  problem  cannot  be  solved  by  SDP.  The  problem  is  that  both  the 
maximum  time  and  the  penalty  are  global  measures.  The  combination  of  the 
two  given  by  the  utility  function  cannot  be  broken  down  into  the  stage-by- 
stage  optimization  problems  needed  by  SDP. 

However,  suppose  the  performance  criterion  is  given  instead  by 

U(E(max  T  ),  E(f(D))),  where  U  is  some  nonlinear  function  and  E  is 
j 

the  expectation  operator.  Now  the  problem  is  separable,  and  can  be  solved 
by  SDP.  The  only  requirement  is  that  the  function  f  be  of  the  nested 
form  needed  for  SDP,  in  which  the  function  value  at  the  current  node  depends 
only  on  the  variables  at  the  current  node  and  the  function  value  at  the 
previous  node.  Examples  of  such  functions  are  the  sum,  product,  or  maximum 
of  the  penalties. 

Figure  2.10  Illustrates  the  situation.  There  are  a  finite  number  of 
allocations,  each  of  which  incurs  an  expected  completion  time  and  an 
expected  penalty.  Indifference  curves  can  be  drawn  using  the  U  function 
to  give  the  tradeoff  between  time  and  penalty.  The  allocation  on  the  lowest 
indifference  curve  is  the  optimal  solution. 


Expected 
Completion 
T  ime 


Figure  2.10.  Time/Penalty  Tradeoff 
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In  the  determinist tc  case,  cf.  [17],  each  decision  variable  x.  .  cook. 

ij 

one  of  two  values,  0  or  1,  depending  on  whether  or  not  task  i  was  allocated 
to  processor  j.  Now  we  need  to  distinguish  between  primary  and  secondary 
assignments.  The  decision  variables  will  now  take  one  or  four  values, 
which  for  simplicity  will  be  denoted  0,  1,  2,  and  3. 

x„  =  0  if  task  i  is  never  assigned  to  processor  j . 

=  1  if  processor  j  is  the  primary  assignment  for  task  i, 
and  there  is  a  secondary  assignment. 

=  2  if  processor  j  is  che  secondary  assignment  for  task  i. 

=  3  if  processor  j  is  the  primary  assignment  for  task  i, 
and  there  is  no  secondary  assignment. 

The  reason  for  distinguishing  between  the  values  1  and  3  is  that  if  processor 
j  fails  and  x  =  1  no  penalty  is  incurred,  but  if  x^ .  =  3  a  penalty 
is  incurred. 

Define  S. .  as  the  set  of  tasks  for  which  x. .  =  k.  If  processor  j 
ij  xj 

does  not  fail,  its  expected  finishing  time  is 


T.  =  l  .  t.  .  + 
J  i  c  S1  1J 

Sij 


Z  ,  t .  .  +  p 

ieS3.  1J 
ij 


leS« 


where  t  .  is  the  execution  time  of  task  i  on  processor  j .  The  expected 

xj  3 

penalty  is  pf(S_). 


The  allocation  problem  posed  above  still  cannot  be  directly  solved 
by  SDP.  Instead  we  solve  the  problem 
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Minimize  max  T. 

j  J 


Subject  to  f(D)  <  2 


and  memory  constraints 


★  * 

for  several  values  of  Z  .  For  each  value  of  Z  ,  call  the  solution  to 

k  k  k 

this  problem  I  .  Several  pairs  (T  ,  Z  )  are  obtained,  and  can  be 
evaluated  using  the  U  function.  The  allocation  which  gives  the  optimal 
pair  is  the  solution. 


In  order  tc  u  e  SDP,  another  state  variable  Z  is  appended.  Z  is 
discretized  usin'  relevant  values  of  the  expected  penalty.  At  each  stage 
the  problem  of  minimizing  the  finishing  time  is  solved,  but  subject  to  the 
penalty  being  no  more  than  each  value  of  Z.  At  the  final  stage  the  values 

k 

of  Z  become  those  of  Z  ,  and  the  comparison  of  solutions  can  be  done. 


The  computations  needed  for  this  problem  are  significantly  more  than 
for  the  deterministic  problem.  To  begin  with,  the  fact  that  the  number 
of  values  for  each  decision  variable  is  increased  from  2  to  4  essentially 
squares  the  number  of  computations  at  each  stage.  The  addition  of  anotner 
state  variable  (the  penalty  level)  also  increases  the  computations.  However, 
if  che  SDP  procedure  is  used  only  in  the  design  process  to  find  an  optimal 
allocation,  this  may  not  be  too  burdensome. 
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SECTION  III 

OPTIMAL  ARCHITECTURES  FOR  DISTRIBUTED  SYSTEMS 

As  stated  in  the  introduction,  SCT's  long-run  goal  is  to  develop  a 
methodology  to  simultaneously  solve  the  software  allocation  and  hardware 
design  problems.  Our  first  step  towards  this  goal  is  to  look  at  che  two 
problems  separately.  A  method  for  integrating  the  two  problems  ill  be 
developed  later. 

This  section  on  optimal  architectures  is  organized  as  follows. 

Section  3.1  presents  an  overview  of  the  problem.  Section  3.2  describes 
a  computer  program  which  was  developed  as  a  tool  to  evaluate  candidate 
architectures.  Section  3.3  looks  at  a  particularly  important  problem 
which  involves  small  interprocessor  distances.  Section  3.4  examines 
several  problems  related  to  network  reliability.  The  final  section  (3.5) 
discusses  the  topic  of  bus  connection  networks,  which  are  a  more  general 
and  more  realistic  way  of  describing  a  distributed  computer  system. 

3.1  PROBLEM  OVERVIEW 

In  this  section  we  consider  the  problem  of  how  to  optimally  design  a 
distributed  system  architecture.  Before  looking  at  various  architectures, 
though,  we  need  to  define  the  system's  building  blocks.  The  main  system 
elements  are  a  set  of  processors.  Each  processor  has  its  own  dedicated 
memory.  The  interprocessor  communication  is  handled  via  a  specified  com¬ 
munication  structure.  In  Sections  3. 1-3. 4  this  structure  consists  of  a 
set  of  bidirectional  links  between  processors  (although  unidirectional 
links  will  also  be  mentioned).  Section  3.5  looks  at  a  more  general  com¬ 
munication  structure,  in  which  groups  of  processors  are  connected  to  buses. 

The  communication  mechanism  will  be  assumed  to  be  a  message-passing 
protocol.  That  is,  if  processor  A  needs  to  send  a  message  to  processor  B, 
the  message  is  sent  through  a  sequence  of  processors  until  it  reaches  B. 
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Either  a  routing  algorithm  or  a  look-up  table  can  be  used  to  determine 
this  sequence.  In  a  local  network  (such  as  an  avionic  system) ,  the  time 
to  send  the  message  along  each  link  is  insignificant  compared  to  the  delay 
incurred  by  having  to  go  through  several  processors.  Thus  the  delay  in¬ 
curred  can  be  assumed  to  be  proportional  to  the  number  of  intermediate 
processors  along  the  route  between  the  given  processors. 

For  a  real  system,  the  delay  depends  on  both  the  specific  hardware 
used  and  the  operating  system.  As  a  result,  this  model  only  pertains  to 
some  systems.  For  example,  in  the  case  of  multiple  processors  connected 
to  a  linear  bus,  the  delay  may  be  proportional  to  the  total  number  of  pro¬ 
cessors  connected. 

A  graph-theoretic  model  has  been  used  to  analyze  various  possible 
architectures.  Such  a  model  uses  only  the  network  topology,  or  how  the 
processors  are  connected,  to  evaluate  how  good  the  network  is.  In  reality, 
there  are  many  other  considerations,  primarily  involving  the  particular 
software  that  is  going  to  be  used  on  the  system.  Nevertheless,  the  topo¬ 
logical  analysis  can  tell  us  quite  a  bit  about  the  network  performance. 

The  following  graph  theory  terms  will  be  frequently  used  in  this 
chapter: 

Note:  a  graphical  representation  of  a  processor,  usually  shown  as 
a  dot  ( • )  • 

Edge:  a  line  segment  connecting  two  nodes,  representing  a  bidirectional 
communication  link  between  two  processors. 

Degree  of  a  node:  the  number  of  edges  connected  to  a  node. 

Degree  of  a  graph:  the  maximum  of  the  degrees  of  the  nodes  in  the 
graph. 

Regular  graph:  a  graph  in  which  every  node  has  the  same  degree. 

Distance  between  two  nodes :  the  minimum  number  of  edges  between 
two  nodes. 

Diameter  of  a  graph:  the  maximum  distance  between  all  pairs  of  nodes. 
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Figure  3.2.  Fully  Connected  Network 
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To  show  how  these  terms  are  used,  consider  a  few  possible  distributed 
architectures,  shown  in  Figures  3.1  -  3.3.  Figure  3.1  shows  a  ring  archi¬ 
tecture  on  6  nodes.  There  are  6  edges  in  the  graph.  Each  node  has  degree  2, 
so  the  graph  is  regular  and  has  degree  2.  The  distance  between  nodes  A 
and  B  is  1,  between  A  and  C  is  2,  and  between  A  and  D  is  3.  The  diameter 
of  the  graph  is  3. 

Figure  3.2  shows  a  fully  connected  architecture  on  6  nodes.  Every  node 
has  degree  5,  so  the  graph  is  regular.  Since  every  pair  of  nodes  is  con¬ 
nected  by  a  link,  the  distance  between  each  pair  is  1.  Thus  the  graph's 
diameter  is  1. 

Figure  3.3  shows  a  hierarchical  architecture  with  7  nodes.  This  graph 
is  not  regular,  since  the  top  node  has  degree  2,  the  next  two  nodes  have 
degree  3,  and  the  bottom  nodes  have  degree  1.  The  diameter  is  4. 


Figure  3.3.  Hierarchical  Network 


In  designing  a  processor  interconnection  network,  there  are  many 
different  factors  to  consider.  These  factors  include  the  following: 
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1)  The  number  of  connections  per  processor  should  be  small.  This  is 
motivated  by  feasibility  and  cost;  processors  with  a  large  number 
of  ports  may  not  be  available,  or  may  be  very  costly. 

2)  The  distance  between  processors  should  be  small.  In  a  message¬ 
passing  network,  the  delay  time  for  a  message  is  approximately 
proportional  to  the  number  of  links  which  must  be  traveled  over 
between  processors.  The  minimization  of  delays  should  be  one  of 
the  objectives  of  the  network  design. 

3)  No  processor  or  link  should  be  on  a  high  proportion  of  the  shortest 
paths  between  processors.  Such  a  configuration  would  cause  queueing 
delays,  which  would  degrade  the  performance. 

4)  The  network  should  be  highly  reliable.  If  a  processor  or  link 
fails,  the  network's  performance  should  not  be  seriously  worsened. 

These  various  factors  are  often  in  conflict.  For  example,  if  we  allow 
each  processor  to  be  connected  to  only  two  others,  then  a  ring  architecture 
(Figure  3.1)  is  the  best  possible.  With  n  nodes,  the  diameter  of  this 
graph  is  n/2.  A  fully  connected  network  (Figure  3.2)  has  a  diameter  of 
only  one,  but  requires  each  of  n  processors  to  be  connected  to  n  -  1  others. 

Figure  3.4  illustrates  the  tradeoff  between  the  two  factors  of  degree 
and  diameter  in  the  case  where  we  have  12  nodes.  As  the  allowable  degree 
increases,  the  possible  diameter  decreases.  Some  graphs  which  correspond 
to  these  values  are  shown  in  Figure  3.5.  The  best  graph  among  these  depends 
upon  the  desired  cost/performance  tradeoff. 

Suppose  the  cost  of  a  network  with  n  processors  depends  solely  on  the 
number  of  links  in  the  network.  Let  us  assume  that  each  processor  can 
accommodate  as  many  as  n-1  links.  If  the  links  are  very  expensive,  the 
best  network  will  have  as  few  links  as  possible.  Assuming  that  the  network 
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must  be  connected  (there  must  be  a  path  between  every  pair  of  nodes),  the 
best  network  would  be  a  tree,  such  as  that  in  Figure  3.3  The  tree  with 
the  smallest  diameter  is  a  star,  such  as  the  one  in  Figure  3.6.  A  major 
problem  with  this  type  of  network  is  that  all  messages  between  processors 
must  go  through  the  central  processor.  Furthermore,  if  the  central  processor 
fails  the  network  becomes  disconnected. 

Clearly  there  are  many  graph  theory  questions  which  are  of  interest 
in  the  design  of  distributed  computer  systems.  The  next  section  describes 
a  tool  which  was  developed  in  order  to  look  at  some  of  these  questions. 


Figure  3.6.  Star  Network 


43 


NADC-81105-50 


3.2  A  NETWORK  EVALUATION  TOOL 

In  order  to  evaluate  a  proposed  network  architecture,  a  computer  program 
called  NETEV  has  been  developed.  The  purpose  of  this  program  is  to  serve  as 
a  design  tool.  Parameters  which  characterize  the  network  are  inputs  to  the 
program.  The  outputs  specify  the  performance  of  the  network,  both  with  and 
without  hardware  failures.  A  listing  of  this  program  appears  as  Appendix  B 
of  this  report.  It  is  written  in  FORTRAN,  for  a  DEC  VAX  11/780. 

The  program  inputs  are: 

—  Number  of  processors 
—  Number  of  links  between  processors 
—  Where  the  links  are 

—  "Distance"  of  links  (which  need  not  be  one) 

The  program  outputs  are: 

—  Maximum  inter-processor  distance  (diameter) 

—  Average  inter-processor  distance 

—  Expected  diameter,  given  a  single  processor  failure 
—  Expected  average  distance,  given  a  single  processor  failure 
—  Expected  diameter,  given  a  single  link  failure 
—  Expected  average  distance,  given  a  single  link  failure 

Fraction  of  links  which  can  fail  and  leave  the  network  connected 
—  Average  fraction  of  connected  processors,  given  a  single  link  failure 

The  code  assumes  that  the  links  are  bidirectional.  However,  the  algorithm 
does  not  require  this,  and  a  simple  modification  of  the  code  would  allow  uni¬ 
directional  links. 

The  first  two  outputs  characterize  the  network  performance  if  there  are 
no  hardware  failures.  To  analyze  the  effect  of  failures,  we  assume  a  model 
in  which  the  probability  that  a  processor  or  link  fails  is  small,  so  that 
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the  probability  of  more  than  one  failure  can  be  safely  neglected.  Further¬ 
more,  we  assume  that  every  processor  fails  with  equal  probability,  and  that 
every  link  fails  with  equal  probability.  The  program  can  be  easily  modified 
so  that  the  probability  that  each  processor  or  link  fails  is  an  input. 

There  are  a  number  of  different  measures  of  the  effect  of  hardware 
failures  on  network  performance.  We  can  analyze  how  the  diameter  or  average 
distance  is  increased  as  a  result  of  a  node  or  link  failure.  However,  for 
many  networks  (such  as  hierarchical  ones)  these  values  will  be  infinity,  and 
so  little  information  is  gained  on  how  reliable  the  network  is.  The  last 
two  measures  attempt  to  rectify  this  problem.  In  some  networks  it  may  be 
important  for  all  processors  to  communicate.  Thus  the  fraction  of  links 
which  can  fail  and  yet  leave  the  network  connected  is  relevant.  In  a  ring 
architecture  any  link  can  fail  and  the  network  will  still  be  connected; 
while  in  a  hierarchical  architecture  if  any  link  fails  the  network  will  be 
disconnected.  However,  perhaps  not  all  pairs  of  processors  are  required 
to  communicate.  As  a  result,  we  may  want  to  know  about  what  fraction  of 
the  processors  can  communicate,  given  a  link  failure. 

The  heart  of  the  program  is  the  calculation  of  the  distance  between 
all  pairs  of  processors.  This  is  done  by  using  the  Floyd-Warshall 
algorithm  [24],  which  is  an  efficient  method  for  finding  the  distance 
between  all  pairs  of  nodes  in  a  graph.  This  algorithm  is  used  on  the  net¬ 
work  for  each  possible  node  or  link  failure. 

As  an  example  of  how  the  method  works,  consider  the  two  graphs  shown 
in  Figure  3.7.  These  graphs  are  examples  of  the  chordal  rings  of  Arden 
and  Lee  [2].  Notice  that  in  the  left  graph  each  chord  (the  links  inside 
of  the  ring)  subtends  5  nodes,  while  in  the  right  graph  each  chord  subtends 
7  nodes.  To  determine  which  of  these  is  optimal  we  can  use  NETEV  to  calcu¬ 
late  the  maximum  distance  and  average  distance.  Assuming  that  each  link 
has  length  1,  the  two  graphs  have  identical  diameters  (4)  and  average 
distances  (2.5263).  However,  suppose  we  can  use  links  with  a  different 
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Figure  3.7.  Two  Chordal  Rings 


bandwidth  for  the  chords.  Using  NETEV,  we  find  that  if  the  bandwidth 
of  the  chord  links  is  greater  than  that  of  the  ring  links,  then  the  left 
graph  is  optimal;  while  if  it  is  less,  the  right  graph  is  optimal.  This 
result  corresponds  to  our  intuition.  If  the  chord  bandwidth  is  low,  the 
chords  must  subtend  many  nodes  in  order  to  be  on  a  shortest  path;  while 
if  the  chord  bandwidth  is  high  shorter  jumps  can  be  used.  NETEV  can  also 
be  used  to  compare  the  performance  with  failures,  but  in  this  case  the 
two  graphs  perform  identically. 

NETEV  allows  us  to  compare  networks  using  many  different  criteria. 

If  we  limit  ourselves  to  only  a  few  criteria,  some  interesting  quantitative 
results  can  be  obtained,  as  shown  in  the  next  section. 

3.3  MINIMAL  DISTANCE  PROBLEM 

Two  of  the  fundamental  quantities  related  to  a  distributed  computer 
system  are  the  cost  and  the  performance.  From  a  graph-theory  viewpoint, 
the  cost  (for  a  fixed  number  of  nodes)  may  be  represented  by  the  number 
of  edges.  The  measurement  of  performance  is  somewhat  more  difficult. 
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Without  knowledge  of  the  software  to  be  used  on  the  system,  we  need  to 
be  concerned  about  the  distance  between  all  pairs  of  nodes.  It  is  diffi¬ 
cult  to  work  with  the  whole  set  of  distances,  so  various  statistics  about 
the  distances  are  used.  Two  commonly  used  statistics  are  the  average  dis¬ 
tance  and  the  maximum  distance,  which  equals  the  network  diameter. 

Our  work  has  principally  focused  on  the  diameter,  for  several  reasons: 

-  The  diameter  characterizes  the  worst-case  situation,  which  is  likely 
to  be  the  one  of  most  concern. 

-  Networks  with  small  diameters  tend  to  have  small  average  distances, 
while  the  converse  is  not  necessarily  true. 

The  diameter  is  quicker  to  use  in  computations.  Given  two  graphs, 
if  the  second  has  a  pair  of  nodes  which  is  further  apart  than  the 
diameter  of  the  first  graph,  then  the  second  graph  has  a  larger 
diameter.  To  compare  average  distances,  all  of  the  distances  in 
the  second  graph  would  have  to  be  calculated. 

Many  interesting  questions  arise  from  the  cost/performance  tradeoff. 

For  example,  given  a  number  of  nodes  n  and  an  allowed  diameter  k,  what 
graph  has  the  fewest  edges?  If  k  =  1,  we  must  have  a  complete  graph 
(Figure  3.2)  with  n(n-l)/2  edges.  If  k  =  2,  then  a  star  network 
(Figure  3.6)  can  be  used,  with  n-1  edges.  For  any  larger  allowed  diameter, 
we  must  still  have  n-1  edges,  in  order  for  the  network  to  remain  connected 
and  the  diameter  to  remain  finite. 

A  problem  with  both  the  complete  graph  and  the  star  graph  is  that  at 
least  one  node  must  have  degree  n-1.  Real  processors  can  be  connected  to 
only  a  limited  number  of  other  processors.  Thus  there  should  be  a  bound 
on  the  degree  of  a  node. 
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The  following  problem  was  posed  by  Bollobas  [  9  ] :  Given  a  number  of 
nodes  n,  a  maximal  degree  d,  and  a  maximal  diameter  k  find  a  graph  with 
Che  fewest  edges.  Unfortunately,  few  results  for  this  problem  can  be  ob¬ 
tained  except  in  limiting  cases.  The  maximum  number  of  edges  under  these 
conditions  is  nd/2,  which  would  occur  if  every  node  had  degree  d.  Will 
this  number  of  edges  be  sufficient  to  produce  a  graph  with  the  desired 
diameter?  If  k  is  too  small,  it  will  not  be.  However,  if  k  n/2,  for 
any  d  2  there  will  be  a  graph  (a  ring) .  Thus  there  will  be  some  minimal 
diameter  for  which  there  exists  a  graph  with  n  nodes  and  degree  d.  We 
seek  this  minimal  diameter,  for  it  gives  us  a  bound  on  the  network  perfor¬ 
mance.  This  minimal  diameter  network  is  likely  to  be  a  regular  graph,  and 
as  such  it  may  have  more  links  than  necessary  for  a  real  system.  However, 
to  design  a  real  system  we  could  start  with  such  a  graph  and  then  remove 
unnecessary  links. 

Thus  the  problem  we  are  interested  in  is:  Given  a  number  of  nodes  n, 
and  a  maximum  degree  d,  find  a  graph  with  the  minimum  diameter  k.  In 
general,  as  the  number  of  nodes  increases  the  minimum  diameter  increases. 
Thus  we  can  consider  a  dual  problem:  Given  a  maximum  degree  d  and  a 
diameter  k,  find  a  graph  with  the  maximum  number  of  nodes.  Of  course, 
when  designing  a  syscem  we  are  not  really  going  to  choose  an  alternative 
with  the  most  processors.  Rather,  the  solution  to  the  dual  problem  will 
say  that  for  a  degree  d  and  diameter  k,  we  can  have  as  many  as  n  nodes. 
If  we  want  to  construct  a  network  with  n'  <  n  nodes,  we  can  probably 
find  one  with  diameter  k  or  less. 

The  maximum  number  of  nodes  in  a  graph  with  degree  d  and  diameter  k 
can  be  denoted  n(d,k).  An  upper  bound  on  n(d,k)  is  easily  calculated. 
From  any  given  node  at  most  d  nodes  can  be  reached  in  a  distance  of  one 
and,  for  j  >  1,  at  most  d(d-l)^  ^  nodes  can  be  reached  in  a  distance 
of  j .  Thus 

n(d,k)  <_  1  +  d  +  ...  +  d(d-l)k_1 

k  (1) 

d(d-l)K  -  2 

d  -  2 
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Expression  1  is  called  the  Moore  bound,  and  any  graph  which  has  chat  number 
of  nodes  is  called  a  Moore  graph.  Most  Moore  graphs  fall  into  two  classes: 
1)  rings  with  an  odd  number  of  nodes,  where  d  =  2;  2)  fully  connected  net¬ 
works,  where  k  =  1.  In  [  31]  it  was  shown  that  for  k  =  2  there  are  only 
a  few  other  Moore  graphs:  the  Petersen  graph  (Figure  3.8)  where  d  =  3; 
the  Hoffman-Singleton  graph,  where  d  =  7;  and  possibly  a  graph  with  d  =  57 
In  [5  ]  and  [15]  it  was  shown  that  there  are  no  other  Moore  graphs.  In  [ 6  ] 
it  was  shown  that  except  for  the  square  there  are  no  graphs  with  a  number 
of  nodes  equal  to  one  less  than  the  Moore  bound.  No  better  upper  bound  on 
n(d,k)  has  been  established. 


Figure  3.8.  Petersen  Graph 


We  shall  denote  by  b(d,k)  a  lower  bound  on  n(d,k).  Values  of 
b(d,k)  can  be  obtained  by  exhibiting  a  graph  with  degree  d,  diameter  k, 
and  b(d,k)  nodes.  A  number  of  authors  have  written  papers  on  finding 
improved  values  of  b(d,k).  These  authors  include  Elspas  [21],  Akers  [  1], 
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Friedman  [25] »  Korn  [35].  Storvick  [44],  Arden  and  Lee  [  3  ],  and  Leland 
et  al  [37].  In  each  of  these  papers,  a  new  construction  method  is  given. 
Some  of  these  methods  are  shown  in  Figures  3.9  -  3.11.  Figure  3.9  shows 
a  "star  polygon"  which  was  used  by  Elspas.  In  Figure  3.10  a  "hinging" 
graph  is  shown.  It  can  be  thought  of  as  three  hierarchical  graphs,  with 
the  nodes  on  the  lowest  levels  joined  together.  This  method  was  used  by 
Friedman  and  Korn,  and  a  generalization  of  it  was  used  by  Storwick.  Arden 
and  Lee  used  a  "multi-tree  structured  network"  approach,  an  example  of 
which  is  shown  in  Figure  3.11.  Leland  used  a  heuristic  method  to  construct 
his  graphs,  which  are  the  largest  published  graphs  for  small  degrees  and 
diameters. 

SCT  has  discovered  a  new  class  of  graphs  which  are  larger  than  those 
of  the  other  authors  for  many  degrees  and  diameters.  These  are  called 
chordal  ring  graphs,  and  are  a  generalization  of  a  structure  proposed  by 
Arden  and  Lee  [2].  Two  examples  of  Arden  and  Lee's  chordal  rings  are 
shown  in  the  previous  section.  Figure  3.7.  In  their  structure,  every  node 
has  degree  3.  The  graph  begins  as  a  ring  on  n  nodes,  then  chords  are 
drawn  connecting  additional  pairs  of  nodes.  Each  odd  node  is  connected 
to  the  even  node  which  is  w  nodes  ahead  of  it  on  the  ring,  where  w  is 
some  specified  odd  number. 

The  generalization  is  twofold.  First,  more  complex  chordal  connection 
schemes  are  used.  The  Arden  and  Lee  rings  can  be  thought  of  as  having  a 
pattern  of  length  two,  in  that  every  other  node  is  connected  to  the  node  w 
nodes  ahead  of  it.  This  can  be  generalized  to  larger  pattern  lengths,  or 
orders .  Figure  3.12  shows  a  chordal  ring  of  order  three,  in  which  every 
third  node  is  connected  to  the  node  across  the  ring  from  it,  while  the  other 
nodes  are  connected  to  nodes  which  are  8  nodes  either  ahead  of  or  behind 
them.  This  graph  has  diameter  4 . 

For  a  given  number  of  nodes,  finding  the  optimal  connection  scheme 
usually  requires  a  computer  search.  For  each  feasible  order  (the  order 
must  divide  the  number  of  nodes),  all  combinations  of  chord  lengths  must 
be  checked.  Letting  r  be  the  order  and  n  the  number  of  nodes,  the 

/  9 

number  of  combinations  has  the  form  c(r)*(n/r)  ,  where  c(r)  is  a 
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Figure  3.9.  Star  Polygon 


Figure  3.10.  Hinging  Graph 
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function  which  grows  like  r! .  (See  [20]  for  this  derivation).  As  either 
n  or  r  grows  large,  checking  all  possibilities  becomes  infeasible.  For 
most  of  Che  results  given  here,  a  random  search  method  was  used,  examining 
a  fixed  number  (about  1000)  of  possibilities. 

The  second  generalization  is  that  larger  degrees  can  be  used  in  chordal 
rings.  Figure  3.13  shows  an  example  of  a  chordal  ring  with  degree  4, 
diameter  3,  and  36  nodes.  While  the  computations  for  finding  optimal  chordal 
rings  are  even  more  burdensome  with  larger  degrees,  those  found  by  the  random 
search  method  are  better  than  graphs  constructed  by  any  other  method. 


For  the  smallest  cases  which  are  unsolved  (degree  3,  diameters  4 
and  5),  heuristic  methods  were  able  to  produce  larger  graphs  than  in  the 
literature.  The  heuristics,  however,  were  based  on  chordal  rings.  Fig¬ 
ure  3.14  shows  the  38  node  diameter  4  graph,  while  Figure  3.15  shows  the 
60  node,  diameter  5  graph.  This  size  of  a  system  is  probably  at  the  upper 
end  of  the  size  of  avionic  systems  in  the  near  future. 

The  status  of  the  problem  of  maximizing  b(d,k)  is  shown  in  Table  3.1, 
for  degrees  and  diameters  not  exceeding  7.  For  some  cases  (those  circled), 
the  largest  possible  graph  has  been  found.  In  the  other  cases,  the  value 
of  n(d,k)  is  an  open  question.  For  those  cases  in  which  there  are  two 
numbers,  the  top  number  is  the  best  published  result  while  the  bottom 
number  is  the  best  result  obtained  by  SCT.  Notice  that  in  almost  all  cases 
we  have  been  able  to  obtain  better  results. 

In  order  to  construct  larger  networks  (with  several  thousand  nodes), 
different  methods  must  be  used.  While  such  networks  are  unlikely  to  be  used 
for  avionic  systems  in  the  near  future,  they  are  a  natural  outgrowth  of 
the  work  on  small  networks.  Furthermore,  with  the  increasing  miniaturization 
of  processors,  such  networks  may  be  feasible  in  a  few  decades. 

The  paper  by  Imase  and  Itoh  [33]  describes  a  particularly  interesting 
network  based  on  de  Bruijn  sequences  [16].  For  degree  d  and  diameter  k, 
these  networks  have  (d/2)  nodes,  which  for  d  >  6  and  sufficiently 
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Figure  3.11.  Multi-Tree  Structured  Network 


Figure  3.12  A  Generalized  Chordal  Ring 


Figure  3.13.  A  Degree  A  Generalized  Chordal  Ring 


J 
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Figure  3.1A.  38  Node,  Degree  3,  Diameter  A  Graph 


Figure  3.15.  60  Node,  Degree  3,  Diameter  5  Graph 
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Circled  Numbers:  Graphs  have  been  proven  to  be  maximal. 

Where  there  are  two  numbers:  Top  numbers  are  from  [15],  bottom  numbers  are  SCT's  results. 
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large  k  is  larger  chan  any  other  known  type  of  network.  These  networks 
have  a  simple  routing  algorithm.  Furthermore,  if  a  processor  failure  occurs 
a  new  route  can  easily  be  calculated  [43].  The  networks  presented  by  Imase 
and  Itoh  can  also  be  unidirectional,  in  which  case  they  are  almost  as  large 
as  theoretically  possible.  The  routing  algorithm,  and  how  to  adjust  it 
when  there  are  failures,  is  given  in  Appendix  C. 

The  other  principal  method  for  constructing  large  networks  is  by  using 
graph  products,  some  of  which  are  described  in  [37]  and  [18].  Products  of 
various  small  graphs  are  used  to  produce  large  graphs.  Products  of  the  new 
chordal  ring  graphs,  using  the  methods  in  [37],  can  produce  larger  graphs 
than  the  examples  cited  in  that  paper. 


While  the  problem  of  maximizing  the  number  of  nodes  in  a  graph  has  not 
been  solved,  substantial  progress  has  been  made.  Of  course,  there  are 
several  other  factors  we  are  interested  in.  The  issue  of  network  reliability, 
and  the  tradeoff  between  this  and  short  distances  for  hierarchical  graphs , 
is  discussed  in  the  following  section. 

3.4  NETWORK  RELIABILITY 

There  are  many  attributes  besides  short  distances  that  are  of  interest 
when  designing  a  network.  This  section  briefly  explores  one  of  these  attributes, 
that  of  network  reliability.  In  particular,  we  look  at  the  tradeoff  between 
reliability  and  short  distances. 

Network  reliability  can  be  characterized  in  many  different  ways.  We 
will  consider  a  simple  model  in  which  each  node  or  edge  fails  with  a  known 
probability.  These  failures  will  be  assumed  to  be  independent;  the  dependent 
case  is  much  harder. 

Some  of  the  important  measures  of  reliability  are: 

-  The  probability  that  a  particular  node  pair,  or  set  of  node  pairs, 
can  communicate. 

-  The  probability  that  all  node  pairs  can  communicate. 
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-  The  probability  that  all  nodes  can  communicate  with  one  particular  node. 

The  expected  number  of  nodes  which  can  communicate  with  a  particular 
node. 

-  The  expected  number  of  node  pairs  which  can  communicate. 

A  few  of  these  are  similar  to  the  quantities  which  can  be  computed  using 
the  NETEV  program  discussed  in  Section  3.2.  However,  that  program  assumed 
that  the  failure  probabilities  were  small,  so  that  the  probability  that  more 
than  one  hardware  element  fails  could  be  safely  neglected.  If  this  assumption 
is  not  valid,  the  problem  becomes  much  more  difficult.  Reference  [47]  pre¬ 
sents  an  algorithm  for  computing  network  reliability,  and  gives  several 
references  to  other  algorithms. 

Bollobas  [  9  ]  discussed  a  problem  which  ties  together  performance,  cost, 
and  reliability.  In  general,  the  failure  of  a  node  or  edge  increases  the 
graph  diameter.  The  problem  he  considered  was  given  a  number  of  nodes  n, 
degree  d,  and  diameter  k,  find  the  graph  with  the  fewest  edges  in  which 
the  deletion  of  any  node  does  not  increase  the  diameter  beyond  k'  >  k. 

Edge  deletion  instead  of  node  deletion  can  also  be  considered.  These  problems 
are  even  more  difficult  than  the  similar  problem  without  failures  discussed 
in  Section  3.3,  and  there  are  even  fewer  results.  Figure  3.16  shows  an 
example  of  a  critical  graph.  If  an  edge  is  deleted,  the  diameter  may  in¬ 
crease  from  6  9. 

Another  measure  of  reliability  which  may  be  easier  to  calculate  is  the 
number  of  nodes  or  edges  which  can  fail  (alone) ,  and  yet  all  remaining  node 
pairs  can  communicate.  Again,  this  may  be  a  relevant  measure  only  if  the 
probability  of  multiple  failures  is  very  small.  There  is  an  interesting 
tradeoff  between  this  measure  and  having  short  distances,  which  we  will  now 
examine. 

In  particular,  suppose  we  have  a  graph  to  which  we  want  to  add  an 
additional  edge  in  an  optimal  fashion.  Such  a  step  might  be  an  iteration 
in  a  network  construction  algorithm,  or  it  might  be  a  separate  item  of 
interest.  The  optimal  place  to  add  it  depends  upon  the  function  we  want 
to  optimize,  as  can  be  seen  from  a  simple  example.  Suppose  we  have  a  large 
number  (several  dozen)  nodes  in  a  linear  architecture  (Figure  3.17).  Let 
us  find  the  optimal  place  to  add  one  additional  edge,  in  order  to  minimize 
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Figure  3.17.  Linear  Architecture 


the  average  interprocessor  distance.  Intuitively  the  edge  should  be 
symmetric,  connecting  processors  at  a  fraction  f  from  both  ends.  Then 
we  can  calculate  that  with  n  processors  the  average  interprocessor 
distance  is  approximately 

(2f3/3  +  f2  -  f/2  +  1/4)  n  . 

The  minimum  value  of  this  function  occurs  at  f  =  (/2  -  l)/2.  Thus  the 
edge  should  connect  processors  about  20%  from  each  end. 

However,  suppose  we  want  to  add  an  edge  in  order  to  increase  the 
network's  reliability.  Let  us  define  the  reliability  as  the  fraction  of 
nodes  (or  edges)  which,  if  they  fail,  will  not  disconnect  the  network. 
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For  the  linear  architecture  example,  this  is  equal  to  1  -  2f.  The  minimum 
of  this  is  at  f  *  0,  where  the  result  is  a  ring  architecture.  Figure  3.18 
plots  the  average  distance  versus  the  reliability.  It  is  optimal  to  be  on 
the  right  side  of  the  curve,  if  high  reliability  and  small  distances  are 
desirable.  Here  f  is  between  0  and  (/2  -  l)/2.  However,  the  best 
point  depends  on  the  tradeoff  between  these  two  factors. 

AVERAGE 


Figure  3.18.  Average  Distance  vs.  Reliability 
for  Link  Added  to  Linear  Graph 

We  can  also  look  at  the  problem  of  finding  the  best  place  to  add  an 
edge  to  a  tree,  or  hierarchical,  network,  for  performance  or  reliability 
purposes.  Cockayne.  Ruskey,  and  Thomason  [14]  look  at  the  problem  of 
finding  the  best  place  to  add  an  edge  to  a  tree  network  in  order  to  minimize 
the  average  distance.  Their  algorithm  enumerates  all  node  pairs  and  calcu¬ 
lates  how  much  the  average  distance  is  improved  by  adding  an  edge  between 
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chose  nodes.  This  is  done  in  an  intelligent  manner,  so  that  the  number  of 

2 

calculations  is  only  proportional  to  n  k,  where  n  is  the  number  of  nodes 
and  k  is  the  diameter.  A  brute  force  method  would  require  calculations 
proportional  to  n^.  Additional  details  on  their  method  appear  in  Appendix  D. 

If  we  define  reliability  by  the  number  of  nodes  or  edges  which  can  fail 
(singly)  and  keep  the  network  connected,  it  turns  out  to  be  relatively  simple 
to  find  the  best  place  to  add  edges  to  a  hierarchical  graph.  Define  a  node 
or  an  edge  to  be  protected  if,  when  it  fails,  the  network  remains  connected. 
For  a  node  of  degree  d  to  be  protected,  the  d  parts  of  the  graph  con¬ 
nected  to  it  must  be  connected  to  each  other.  This  requires  at  least  d-1 
additional  edges.  For  an  edge  to  be  protected,  it  must  lie  in  a  cycle 
(that  is,  the  two  nodes  which  the  edge  connects  must  have  another  path  be¬ 
tween  them) . 

The  optimal  method  can  best  be  illustrated  by  an  example.  Figure  3.19a 
shows  a  hierarchical  graph,  which  is  "regular"  in  the  sense  that  every  node 
above  the  bottom  row  has  the  same  number  of  successors.  The  method  works 
only  on  networks  of  this  type.  The  basic  idea  is  to  protect  nodes  and 
edges  from  the  top  of  the  hierarchy  to  the  bottom,  protecting  as  many  as 
possible  at  each  step.  In  the  example,  the  bottom  nodes  are  originally 
protected,  but  no  edges  are.  In  order  to  protect  the  top  node  at  least  3 
edges  must  be  added,  while  in  order  to  protect  each  second  row  node  at  least 
4  edges  must  be  added.  The  largest  cycles  which  can  be  created  use  4  of  the 
original  edges.  Figure  3.19b  shows  how  two  new  edges  protect  8  old  ones 
(the  dark  ones).  With  one  more  edge  the  top  node  is  protected  and  circled, 
as  well  as  2  more  edges.  This  is  shown  in  Figure  3.19c.  With  2  more  edges, 
as  in  Figure  3.19d,  two  more  nodes  and  4  more  edges  are  protected.  The  final 
two  nodes  and  6  edges  can  be  protected  with  3  new  edges,  as  in  Figure  3.19e. 
Thus  8  edges  were  needed  to  protect  every  hardware  element.  Clearly  fewer 
are  insufficient,  as  each  new  edge  can  protect  at  most  two  of  the  bottom 
row  of  edges. 
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b.  Addition  of  two  edges  to  proO':t  8  edges 


Figure  3.19.  Additional  Edges  to  Increase  Reliability 
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e.  Three  more  edges  to  protect  2  nodet  6  edges 


Figure  3.19  (Continued).  Addition  of  Edges  to  Increase  Reliability 
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We  have  seen  many  types  of  networks  with  links  which  connect  two  nodes. 
A  more  general  communication  structure  may  be  more  faithful  to  real  systems, 
and  may  produce  better  results.  This  topic  is  explored  in  the  next  section. 

3.5  BUS  CONNECTION  NETWORKS 

In  some  computer  systems,  the  model  of  having  two  nodes  connected  by 
an  edge  is  not  very  realistic  for  the  communication  mechanism.  Instead, 
several  processors  may  be  connected  to  a  bus  and  may  be  thought  of  as  being 
equally  distant  from  each  other.  At  the  same  time,  each  processor  may  be 
connected  to  several  buses.  The  result  is  a  bus  connection  network. 

Figure  3.20  illustrates  the  situation.  For  processor  1  to  communicate 
with  processor  5,  a  message  is  sent  through  bus  A  to  processor  3.  This 
processor  then  sends  the  message  along  bus  B  to  processor  5.  Since  two 
buses  were  used  to  send  the  message,  we  can  say  that  the  distance  between 
processors  1  and  5  is  2. 


Figure  3.20.  Example  of  a  Bus  Connection  Network 

We  will  use  a  model  discussed  by  Mickunas  [42 ]  for  a  bus  connection 
model.  In  this  model,  each  node  is  incident  on  a  certain  number  of  buses, 
which  we  will  call  its  degree.  The  maximum  of  the  degrees  of  the  nodes 
will  be  called  the  graph's  nodal  degree.  Each  bus  has  a  certain  number 
of  nodes  on  it,  which  will  be  called  the  degree  of  the  bus.  The  maximum 
of  the  degrees  of  the  buses  will  be  called  the  graph's  bus  degree.  For 
notation,  we  will  denote  the  nodal  degree  by  d,  the  bus  degree  by  b,  and 
the  diameter  by  k. 
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Two  nodes  will  have  a  distance  of  one  if  they  are  incident  on  a  common 
bus.  In  general,  the  distance  between  two  nodes  is  the  minimum  number  of 
buses  which  must  be  passed  through  to  get  between  them. 

The  standard  graphical  representation  of  these  networks  is  to  use 
lines  as  buses,  connecting  the  nodes  which  are  points.  However,  these  "lines 
become  complex  curves  in  even  very  small  networks,  making  any  sort  of  visual 
analysis  impossible.  Instead,  we  will  use  a  representation  of  the  network 
as  a  bipartite  graph.  A  bipartite  graph  has  two  types  of  nodes,  and  nodes 
of  each  type  are  connected  only  to  nodes  of  the  other  type.  One  type  of 
node  represents  the  original  nodes,  while  the  other  type  -epresents  the  buses 
In  the  figures  in  this  section  nodes  are  filled-in  circle  ,  buses  are  empty 
circles.  An  edge  represents  a  node  incident  on  a  bus.  Figure  3.21  compares 
the  two  representations  for  a  particularly  interesting  graph  with  7  nodes 
and  7  buses,  each  with  degree  3.  The  graphs  have  diameter  1.  In  the  first 
figure,  each  side  of  the  triangle,  each  median,  and  the  circle  in  the  middle 
represent  buses.  The  node  correspondences  between  the  two  figures  are  given 
by  letters.  One  particular  advantage  of  this  new  representation  is  that 
many  results  from  standard  graphs  can  be  used. 

A  concept  analogous  to  Moore  graphs  can  be  defined  for  bus  connection 
networks.  From  each  node  at  most  d  buses  can  be  reached,  from  which  at 
most  d(b-l)  nodes  can  be  reached.  Thus  the  maximum  number  of  nodes  in  a 
diameter  1  graph  is  1  +  d(b-l).  Similarly,  in  two  steps  at  most 
d(d-l) (b-l)^  nodes  can  be  reached  and,  in  j  steps,  at  most  d(d-l)^  1 
(b-l)''  nodes  can  be  reached.  If  we  define  n(d,b,k)  as  the  maximum 
number  of  nodes  in  a  graph  with  nodal  degree  d,  bus  degree  b,  and  diameter 
k,  we  have 


n(d,b,k)  <  1  +  d(b-l)  +  d  (d-1)  (b-1) 2  +  ...  +  d  (d-D^-1  (b-l)k 


-  1  +  d(b-l) 


(d-l)k(b-l)k  -  1 
(d-1) (b-1)  -  1  • 


A  graph  with  this  number  of  nodes  is  called  a  Moore  geometry. 
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A 


A 


Figure  3.21.  Two  Representations  of  7  Node,  Diameter  1  Graph 


Since  there  are  very  few  Moore  graphs,  it  is  natural  to  ask  if 

there  are  any  Moore  geometries.  Let  us  first  consider  the  case  of  d  *  1. 

A  Moore  geometry  with  diameter  1  has  1  +  d(b-l)  nodes.  The  simplest 

case  here  is  where  d  =  b  (the  node  degree  and  bus  degree  are  the  same), 

2 

so  there  are  d“  -  d  +  1  nodes,  and  the  same  number  of  buses.  Every 
pair  of  nodes  is  on  exactly  one  common  bus. 

The  existence  of  such  a  graph  depends  on  the  existence  of  a  mathe¬ 
matical  object  called  a  finite  projective  plane,  a  subject  which  has  been 
extensively  examined.  It  can  be  shown  that  such  a  plane  exists  if  d  -  1 
is  a  prime  power  [45]-  In  certain  other  cases  it  can  be  proven  that  a 
projective  plane  does  not  exist  [11].  However,  the  general  problem  remains 
unsolved.  For  values  of  d  between  3  and  10  projective  planes  exist  in 
all  cases  except  d  =  7. 
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Now  let  us  look  at  the  case  of  d  ^  b.  These  graphs  are  known  as 

balanced  incomplete  block  designs,  or  BIBDs.  Such  designs  are  used  in 

constructing  agricultural  and  biological  experiments.  There  are  certain 

restrictions  on  the  parameters  of  a  BIBD.  Since  nd/b  is  the  number  of 

buses,  this  number  must  be  an  integer.  It  can  also  be  shown  that  we  need 

d  b  [27].  While  these  conditions  are  not  sufficient  to  guarantee 

the  existence  of  a  BIBD,  if  d  *  3  or  4  they  are  sufficient  [28].  In 

addition,  if  an  order  d-1  projective  plane  exists,  then  a  BIBD  can  be 

2 

derived  with  the  same  d,  b  =  d-1,  and  (d-1)  nodes.  Several  examples 
of  BIBDs  are  given  in  [23]. 

The  question  of  the  existence  of  Moore  geometries  with  diameters 
greater  than  one  is  more  complicated.  None  are  known  if  b  >  2.  Bose 
and  Dowling  [10]  give  necessary  conditions  for  existence  when  k  =  2, 
although  they  could  find  no  graphs  satisfying  those  conditions.  Fuglister  [26] 
showed  that  there  are  no  Moore  geometries  with  k  =  3. 

One  simple  result  is  that  there  are  no  Moore  geometries  with  d  =  2 
and  b  >  2  (if  b  *  2  we  get  rings).  This  is  an  important  case  since 
many  real  microprocessors  have  two  ports.  The  proof  of  this  is  given  in 
Appendix  E. 

Several  construction  methods  for  bus  connection  networks  have  been 
proposed  in  the  literature.  These  include  the  hypercube  and  dual  bus 
hypercube  [46],  and  snowflake  and  star  graphs  [22].  These  constructions 
are  motivated  by  their  simple  structure  and  easy  routing  algorithms.  If 
we  are  interested  in  small  interprocessor  distances,  however,  these  simple 
structures  may  not  be  the  best. 

Consider  a  problem  analogous  to  the  one  discussed  in  section  3.3  for 
standard  graphs.  We  seek  a  graph  with  the  maximum  number  of  nodes  which 
has  node  degree  d,  bus  degree  b,  and  diameter  k.  To  begin  with,  let 
us  consider  some  small  special  cases. 
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1.  n(l,  b,  1)  »  b 

This  is  obvious  —  it  is  a  single  bus  of  degree  b.  It  is  the  only 
trivial  case  of  a  Moore  geometry  with  b  >  2. 

2.  n(2,  b,  1)  *  b  +  1. 

From  the  graph  in  Figure  3.22  (using  the  bipartite  graph  representa¬ 
tion),  we  see  that  n(2,  b,  1)  ^  b  +  1.  To  show  we  cannot  put  more  nodes 
in  the  same  graph,  consider  an  initial  graph  consisting  of  the  top  node  and 
top  two  buses.  A  node  which  is  on  the  left  bus  must  share  a  bus  with  all 
nodes  on  the  right  bus.  But  each  of  those  nodes  must  share  a  bus  with  all 
nodes  on  the  left  bus.  As  a  result,  all  second-tier  nodes  must  share  a 
common  bus,  so  there  can  be  only  b.  Adding  the  top  node  gives  b  +  1 
total  nodes. 

3.  n(2 ,  b,  2)  =  b2  +  1. 

2 

The  graph  in  Figure  3.23  shows  n(2,  b,  2)  >_  b  +1,  for  b  3  5 .  The 
analogous  construction  for  other  b's  is  obvious.’  To  show  this  is  the 
maximum  possible  number,  look  at  Figure  3.24,  which  shows  the  largest 
potential  graph  from  a  particular  node.  Call  the  nodes  below  a  level  3/2 
bus  a  group.  Every  left  level  2  node  must  share  a  bus  with  some  node  in  each 
right  group,  in  order  to  reach  every  right  level  one  node.  The  converse 
applies  to  each  right  level  2  node.  Thus  every  bus  must  contain  a  repre¬ 
sentative  from  every  group.  As  a  result,  there  can  be  at  most  b  groups. 

2 

But  then  there  are  1  +  b  +  b(b-l)  =  b  +1  nodes. 

In  both  cases  2  and  3  these  graphs,  which  are  maximal,  are  about  half 
as  large  as  the  Moore  bound. 

For  several  values  of  r  and  b,  the  value  of  n(r,  b,  1)  can  be 
determined.  For  example,  n(3,  4,  1)  *  8,  as  shown  in  Figure  3.25.  Several 
other  examples  are  given  in  (19]. 
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•  Nodes 


O  Buses 


Figure  3.22.  Graph  Showing  n(2,  b,  1)  ^  b  +  1 
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Figure  3.24.  Graph  Showing  n(2,  b,  2)  <_  b  +1 


Additional  buses 

connect 

ABEF;  ABDG; 

CEFG;  CD. 


Figure  3.25.  Graph  Showing  n(3,  4,  1)  -  8 
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For  larger  networks,  general  construction  methods  must  be  developed. 

While  there  will  not  produce  maximal  graphs,  they  will  produce  reasonably 
large  ones.  The  basic  method  is  to  use  special  cases  of  the  graphs  described 
in  Section  3.3  which  are  bipartite.  Certain  star  polygons,  multi-tree 
structured  networks,  hingings,  and  chordal  rings  are  bipartite.  One  type 
of  node  is  designated  the  processors,  and  the  other  type  is  designated  the 
buses.  The  diameter  can  then  be  calculated,  and  the  graph  evaluated.  More 
details  on  these  methods  can  be  found  in  [19]. 

Figure  3.26  shows  a  hinging  example.  In  this  type  of  graph  the  processor 
nodes  and  bus  nodes  can  have  different  degrees.  Here  d  *  3,  b  =  4,  and  the 
diameter  k  ■*  3.  There  are  40  processor  nodes. 

Figure  3.27  shows  how  a  chordal  ring  can  represent  a  bus  connection 
network.  In  this  case  the  bus  degree  and  processor  degree  must  be  equal. 

Here  there  are  24  processor  nodes  and  a  diameter  of  2. 

The  graphs  based  on  de  Bruijn  sequences  can  be  generalized  for  use  as 
bus  connection  networks.  These  networks  will  have  a  size  of  (db/4)  ,  when 
d  and  b  are  both  even.  In  addition,  some  of  the  graph  products  can  be 
used  with  bus  connection  networks.  Both  of  these  topics  are  discussed 
in  [19]. 
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Figure  3.26.  A  Hinging  With  r 
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APPENDIX  A 


SPATIAL  DYNAMIC  PROGRAMMING  COMPUTER  CODE 
FOR 

SOFTWARE  ALLOCATION 


A-l 


PROCRAM  SDP2< INPUT, OUTPUT.  INFILE. OUTFILE), 
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APPENDIX  C 

ROUTING  ALGORITHM  FOR  DE  BRUIJN  GRAPHS 

I 

To  describe  Che  routing  algorithm,  the  representation  in  [37]  will 
be  used.  Each  node  is  represented  as  a  vector  of  k  symbols,  each 
taking  values  from  1  to  d/2  (assuming  d  is  even) .  Node 

(vrv2,...,vk) 

is  connected  to  all  nodes  of  the  form 
(x,v1,v2,...,vk_1) 
or 

(v2,v3,.. . ,vk,x) 

where  x  takes  any  value  from  1  to  d/2.  To  show  that  the  graph  has 
diameter  k^  suppose  we  want  to  go  from  (v^ ,v2 , . . . , vk>  to  (w^ ,w2 , . . . ,wk) . 
At  the  first  step  go  to  (wk»v^>v2  ,  •  •  •  ,Vk_^)  ,  at  the  second  step  to 

(VrVV-’-’Vz)’  and  ln  seneral  at  the  tth  steP  co  (V(H)' 
wk_^_2j , .  . .  ,vk  ^) .  After  k  steps  the  desired  node  will  be  reached. 

Suppose  we  want  to  go  from  (v0>vi»  •  *  •  »\_1>  to  (v,  ,v3 , . . .  .v^  )  . 

By  the  above  method  this  would  be  accomplished  in  two  steps,  going  through 
(v^,v2 , . . . ,vk) .  However,  suppose  this  last  node  is  not  functioning. 

A  different  sequence  involving  only  four  additional  steps  can  be  taken  [43]. 
Let  a,b,c,  and  e  take  values  from  1  to  d/2.  The  sequence  is 

STEP  NODE  CONDITIONS 


0 

(v0,Vl’*’ * ,Vk-l) 

1 

(v  ,. •• »vfc-1»a) 

2 

(v2>v3> • • • tvk  ^,a,b) 

3 

(c,v2,.  . .  ,v  ,a) 

4 

(e,c,v2>. . .  .v^) 

e^x^ 

5 

(c , v2 , * . . >^k) 

6 

(v2,. • • .vk.vk+1> 
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This  method  will  work  as 
are  more  failures,  additional 


long  as  there  is  only  one 
conditions  may  have  to  be 


failure . 
imposed . 


If  there 


I 

I 
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APPENDIX  D 

ALGORITHM  FOR  BEST  LINK  ADDITION  IN  A  TREE  NETWORK 

This  algorithm,  from  [14],  finds  the  best  place  to  add  a  link  in  a 
tree  network  in  order  to  minimize  the  average  distance.  All  pairs  of  nodes 
are  considered,  but  the  enumeration  is  organized  so  that  the  time  to  cal¬ 
culate  the  improvement  is  proportional  to  the  graph  diameter. 

The  algorithm  is  given  in  the  paper  as  a  PASCAL  procedure.  Three 
data  structures  are  used.  A  vertex  and  its  associated  information  is  called 
a  node  and  is  defined  by: 

node  =  record 

adj :  t  edge; 
desc :  1. . .n; 
imp,tri,sum:  integer 
end 

The  nodes  are  organized  into  an  array  called  "a",  in  which  they  are  numbered 
from  1  to  n.  The  adjacencies  in  the  network  are  given  by  "edges",  where 

edge  =  record 

vert:  vertex 
next :  + edge 
end. 

The  procedure  is  as  follows: 
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procedure  DFS(w:  vertexjd:  1  . ..n); 

var  p :  +  edge ; 
begin  p  :=a(w j . adj ; 
while  p  i4  nil  do 

with  a(pt.vert]  do  {  w'  apt  .vert; 
begin  t[d]  :=a[w].desc  -  desc; 
if  d  mod  2=1  then 
begin  {  d  is  odd} 
imp  :=a[w].imp  -  a[w].tri 
+  2*desc*a[w] . sum; 
trouble  :=0;i  :=l;j  :=(d+3)div  2; 
while  j<=d  do 

begin 

trouble  :=trouble  +t[i]*t[j]; 
i  :=i+l; j  :=j+l 
end; 

tri  :a[w].tri  -  trouble; 
sum  :=a[w].sum  +  t[(d+l)div  2] 
end 

else  begin  { d  is  even} 

tri  :=a[w].tri  +  t[d  div  2]*desc; 
imp  :=a[w].imp  -  tri  +  2*desc*a[w] .sum; 
sum  :=a[w].sum 
end; 

if  imp  >  max  then 
begin  max  : = imp ; 
best_vertex. vl  :=v 
best_vertex. v2:=p* . vert 
end ; 

DFS (p  t . vert ,d+l ) ; 
p  :=pi.next 
end 
end; 


By  calling  DFS(v,l),  the  best  vertex  to  connect  v  with  becomes  the 
value  of  the  global  variable  best_vertex.v2.  This  can  be  done  for  all 
vertices  v,  and  thus  the  best  pair  is  found.  Methods  are  given  in  the 
paper  for  reducing  the  number  of  computations  even  further. 
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APPENDIX  E 


Theorem:  There  are  no  Moore  geometries  when  the  nodal  degree  is  2  and  the 
bus  degree  is  greater  than  2. 


Proof:  If  such  a  Moore  geometry  did  exist,  it  would  have 

1  +  2 [ (k-1)  +  (k-1)2  +  ...  +  (k-l)d]  (1) 

nodes.  Now  in  a  Moore  geometry  the  number  of  nodes  times  the  nodal  degree 
must  equal  the  number  of  buses  times  the  bus  degree.  Since  the  number  of 
buses  is  integral,  we  must  have  twice  expression  (1)  divisible  by  k.  That 
is,  we  need 

2  +  4 [ (k-1)  +  (k-1)2  +  ...  +  (k-l)d]  =  0  (mod  k) .  (2) 

Most  of  the  terms  on  the  left,  when  the  powers  are  expanded,  are  powers  of 
k  and  can  be  eliminated  without  affecting  congruence  (2).  Thus  we  must  have 
2  +  r [ (-1)  +  (-1)2  +  ...  +  (-l)d]  =  0  (mod  k)  (3) 

If  d  is  even,  (3)  becomes 

2=0  (mod  k) 


while  if  d  is  odd,  (3)  becomes 

-2=0  (mod  k) . 

In  either  case,  k  cannot  be  larger  chan  2.* 


